[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 13:00:31.580368  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:39787
I20260709 13:00:31.582087  7481 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 13:00:31.583586  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:31.603718  7490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:31.603763  7487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:31.605616  7481 server_base.cc:1061] running on GCE node
W20260709 13:00:31.603768  7488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:32.824654  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:32.825002  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:32.825238  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602032825213 us; error 0 us; skew 500 ppm
I20260709 13:00:32.826184  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:32.832337  7481 webserver.cc:533] Webserver started at http://127.7.78.126:37965/ using document root <none> and password file <none>
I20260709 13:00:32.833381  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:32.833724  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:32.834283  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:32.840921  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "949ee267387b4cd2b7b81de073bf36b8"
format_stamp: "Formatted at 2026-07-09 13:00:32 on dist-test-slave-50cs"
I20260709 13:00:32.849929  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.002s	sys 0.005s
I20260709 13:00:32.855489  7496 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:32.856743  7481 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:00:32.857151  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "949ee267387b4cd2b7b81de073bf36b8"
format_stamp: "Formatted at 2026-07-09 13:00:32 on dist-test-slave-50cs"
I20260709 13:00:32.857575  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:32.922029  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:32.923743  7481 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 13:00:32.924260  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:32.997918  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:39787
I20260709 13:00:32.997987  7547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:39787 every 8 connection(s)
I20260709 13:00:33.005999  7548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.028091  7548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8: Bootstrap starting.
I20260709 13:00:33.034032  7548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:33.035954  7548 log.cc:826] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8: Log is configured to *not* fsync() on all Append() calls
I20260709 13:00:33.040712  7548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8: No bootstrap required, opened a new log
I20260709 13:00:33.061781  7548 raft_consensus.cc:359] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ee267387b4cd2b7b81de073bf36b8" member_type: VOTER }
I20260709 13:00:33.062376  7548 raft_consensus.cc:385] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:33.062675  7548 raft_consensus.cc:740] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949ee267387b4cd2b7b81de073bf36b8, State: Initialized, Role: FOLLOWER
I20260709 13:00:33.063673  7548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ee267387b4cd2b7b81de073bf36b8" member_type: VOTER }
I20260709 13:00:33.064224  7548 raft_consensus.cc:399] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:00:33.064538  7548 raft_consensus.cc:493] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:00:33.064911  7548 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:33.069177  7548 raft_consensus.cc:515] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ee267387b4cd2b7b81de073bf36b8" member_type: VOTER }
I20260709 13:00:33.070092  7548 leader_election.cc:304] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 949ee267387b4cd2b7b81de073bf36b8; no voters: 
I20260709 13:00:33.071921  7548 leader_election.cc:290] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:00:33.072316  7551 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:33.074653  7551 raft_consensus.cc:697] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 1 LEADER]: Becoming Leader. State: Replica: 949ee267387b4cd2b7b81de073bf36b8, State: Running, Role: LEADER
I20260709 13:00:33.075695  7551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ee267387b4cd2b7b81de073bf36b8" member_type: VOTER }
I20260709 13:00:33.076818  7548 sys_catalog.cc:565] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:00:33.087692  7552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 949ee267387b4cd2b7b81de073bf36b8. Latest consensus state: current_term: 1 leader_uuid: "949ee267387b4cd2b7b81de073bf36b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ee267387b4cd2b7b81de073bf36b8" member_type: VOTER } }
I20260709 13:00:33.087780  7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "949ee267387b4cd2b7b81de073bf36b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949ee267387b4cd2b7b81de073bf36b8" member_type: VOTER } }
I20260709 13:00:33.088495  7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:33.088495  7552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:33.091555  7560 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:00:33.103223  7560 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:00:33.106400  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:00:33.123281  7560 catalog_manager.cc:1383] Generated new cluster ID: ce05311c38914498b18f1847e7c9cd1a
I20260709 13:00:33.123560  7560 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:00:33.136500  7560 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:00:33.138139  7560 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:00:33.153745  7560 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8: Generated new TSK 0
I20260709 13:00:33.154795  7560 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:00:33.173512  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:33.184064  7570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:33.185237  7571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:33.189177  7481 server_base.cc:1061] running on GCE node
W20260709 13:00:33.189785  7573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:33.190811  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:33.191174  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:33.191375  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602033191351 us; error 0 us; skew 500 ppm
I20260709 13:00:33.192008  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:33.196995  7481 webserver.cc:533] Webserver started at http://127.7.78.65:38975/ using document root <none> and password file <none>
I20260709 13:00:33.197645  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:33.197882  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:33.198171  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:33.199280  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
format_stamp: "Formatted at 2026-07-09 13:00:33 on dist-test-slave-50cs"
I20260709 13:00:33.205245  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20260709 13:00:33.209856  7578 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.210753  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 13:00:33.211071  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
format_stamp: "Formatted at 2026-07-09 13:00:33 on dist-test-slave-50cs"
I20260709 13:00:33.211380  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:33.229434  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:33.230937  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:33.233815  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:33.238601  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:33.238847  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.239171  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:33.239332  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.352666  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:43171
I20260709 13:00:33.352766  7640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:43171 every 8 connection(s)
I20260709 13:00:33.359701  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:33.372430  7645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:33.375948  7646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:33.381469  7648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:33.382220  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:33.383013  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:33.383241  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:33.383425  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602033383405 us; error 0 us; skew 500 ppm
I20260709 13:00:33.383998  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:33.391526  7481 webserver.cc:533] Webserver started at http://127.7.78.66:43115/ using document root <none> and password file <none>
I20260709 13:00:33.392285  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:33.392561  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:33.392930  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:33.393435  7641 heartbeater.cc:344] Connected to a master server at 127.7.78.126:39787
I20260709 13:00:33.393966  7641 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:33.394536  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "b523fc673cf34021a277e15cd85c3b20"
format_stamp: "Formatted at 2026-07-09 13:00:33 on dist-test-slave-50cs"
I20260709 13:00:33.395241  7641 heartbeater.cc:507] Master 127.7.78.126:39787 requested a full tablet report, sending...
I20260709 13:00:33.399264  7513 ts_manager.cc:194] Registered new tserver with Master: a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171)
I20260709 13:00:33.401466  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20260709 13:00:33.402916  7513 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34844
I20260709 13:00:33.408069  7653 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.409226  7481 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:00:33.409628  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "b523fc673cf34021a277e15cd85c3b20"
format_stamp: "Formatted at 2026-07-09 13:00:33 on dist-test-slave-50cs"
I20260709 13:00:33.409983  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:33.428166  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:33.429685  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:33.432502  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:33.435771  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:33.435985  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.436321  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:33.436461  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.538739  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:38459
I20260709 13:00:33.538962  7715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:38459 every 8 connection(s)
I20260709 13:00:33.544905  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:33.554168  7719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:33.559942  7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:33.562878  7716 heartbeater.cc:344] Connected to a master server at 127.7.78.126:39787
I20260709 13:00:33.563352  7716 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:33.564330  7716 heartbeater.cc:507] Master 127.7.78.126:39787 requested a full tablet report, sending...
W20260709 13:00:33.566350  7722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:33.567242  7513 ts_manager.cc:194] Registered new tserver with Master: b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:33.568833  7513 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34848
I20260709 13:00:33.570484  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:33.571534  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:33.571813  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:33.572047  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602033572023 us; error 0 us; skew 500 ppm
I20260709 13:00:33.572765  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:33.575826  7481 webserver.cc:533] Webserver started at http://127.7.78.67:43269/ using document root <none> and password file <none>
I20260709 13:00:33.576457  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:33.576700  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:33.577018  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:33.578613  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "76ae0ee745734d5e9088ff6563d716d8"
format_stamp: "Formatted at 2026-07-09 13:00:33 on dist-test-slave-50cs"
I20260709 13:00:33.584800  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20260709 13:00:33.590320  7727 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.591353  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 13:00:33.591795  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "76ae0ee745734d5e9088ff6563d716d8"
format_stamp: "Formatted at 2026-07-09 13:00:33 on dist-test-slave-50cs"
I20260709 13:00:33.592227  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:33.613893  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:33.615180  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:33.616593  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:33.619772  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:33.620015  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.620266  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:33.620445  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:33.715415  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:37871
I20260709 13:00:33.715553  7789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:37871 every 8 connection(s)
I20260709 13:00:33.735194  7790 heartbeater.cc:344] Connected to a master server at 127.7.78.126:39787
I20260709 13:00:33.735625  7790 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:33.736548  7790 heartbeater.cc:507] Master 127.7.78.126:39787 requested a full tablet report, sending...
I20260709 13:00:33.738868  7513 ts_manager.cc:194] Registered new tserver with Master: 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:33.739913  7481 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017822657s
I20260709 13:00:33.740428  7513 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34852
I20260709 13:00:33.809916  7513 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34856:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:00:33.815096  7513 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:00:33.927165  7606 tablet_service.cc:1511] Processing CreateTablet for tablet f3128520a7c344758ca9dac16999e350 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:00:33.927716  7604 tablet_service.cc:1511] Processing CreateTablet for tablet e91e12d6280746fd8de26cccf5330429 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:00:33.928973  7606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3128520a7c344758ca9dac16999e350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.929682  7604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91e12d6280746fd8de26cccf5330429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.929945  7601 tablet_service.cc:1511] Processing CreateTablet for tablet e5d74af98f944e3ca1510f01c68ab683 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:00:33.931294  7601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d74af98f944e3ca1510f01c68ab683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.927280  7605 tablet_service.cc:1511] Processing CreateTablet for tablet c3b6ec9f23a94abea53c55ef57a0fac6 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:00:33.932474  7605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b6ec9f23a94abea53c55ef57a0fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.933858  7602 tablet_service.cc:1511] Processing CreateTablet for tablet 2f4cb96a3fc044fd92d7be13e0807e60 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:00:33.935427  7602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4cb96a3fc044fd92d7be13e0807e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.938804  7603 tablet_service.cc:1511] Processing CreateTablet for tablet 4e834a2c2ef24568a8505066a5b8bdd3 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:00:33.940356  7603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e834a2c2ef24568a8505066a5b8bdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.943804  7600 tablet_service.cc:1511] Processing CreateTablet for tablet 968f7bdc989246688fc4e10e8fd870a2 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:00:33.945851  7598 tablet_service.cc:1511] Processing CreateTablet for tablet 28f891b1555d4bdf94235c996aef9354 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:00:33.946345  7599 tablet_service.cc:1511] Processing CreateTablet for tablet 1d44fab36da747c08f6c6b34bba33782 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:00:33.948022  7600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 968f7bdc989246688fc4e10e8fd870a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.949263  7598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f891b1555d4bdf94235c996aef9354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:33.957288  7599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d44fab36da747c08f6c6b34bba33782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.005575  7807 tablet_bootstrap.cc:492] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:34.036793  7752 tablet_service.cc:1511] Processing CreateTablet for tablet 4e834a2c2ef24568a8505066a5b8bdd3 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:00:34.036939  7753 tablet_service.cc:1511] Processing CreateTablet for tablet e91e12d6280746fd8de26cccf5330429 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:00:34.034447  7754 tablet_service.cc:1511] Processing CreateTablet for tablet c3b6ec9f23a94abea53c55ef57a0fac6 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:00:34.041039  7751 tablet_service.cc:1511] Processing CreateTablet for tablet 2f4cb96a3fc044fd92d7be13e0807e60 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:00:34.037873  7755 tablet_service.cc:1511] Processing CreateTablet for tablet f3128520a7c344758ca9dac16999e350 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:00:34.045289  7807 tablet_bootstrap.cc:654] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.045091  7750 tablet_service.cc:1511] Processing CreateTablet for tablet e5d74af98f944e3ca1510f01c68ab683 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:00:34.064700  7680 tablet_service.cc:1511] Processing CreateTablet for tablet c3b6ec9f23a94abea53c55ef57a0fac6 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:00:34.064817  7681 tablet_service.cc:1511] Processing CreateTablet for tablet f3128520a7c344758ca9dac16999e350 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:00:34.062163  7752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e834a2c2ef24568a8505066a5b8bdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.056835  7749 tablet_service.cc:1511] Processing CreateTablet for tablet 968f7bdc989246688fc4e10e8fd870a2 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:00:34.072708  7679 tablet_service.cc:1511] Processing CreateTablet for tablet e91e12d6280746fd8de26cccf5330429 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:00:34.070843  7755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3128520a7c344758ca9dac16999e350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.074301  7747 tablet_service.cc:1511] Processing CreateTablet for tablet 28f891b1555d4bdf94235c996aef9354 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:00:34.061889  7748 tablet_service.cc:1511] Processing CreateTablet for tablet 1d44fab36da747c08f6c6b34bba33782 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:00:34.075294  7678 tablet_service.cc:1511] Processing CreateTablet for tablet 4e834a2c2ef24568a8505066a5b8bdd3 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:00:34.084627  7681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3128520a7c344758ca9dac16999e350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.089457  7753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91e12d6280746fd8de26cccf5330429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.094365  7679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91e12d6280746fd8de26cccf5330429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.095752  7677 tablet_service.cc:1511] Processing CreateTablet for tablet 2f4cb96a3fc044fd92d7be13e0807e60 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:00:34.083426  7676 tablet_service.cc:1511] Processing CreateTablet for tablet e5d74af98f944e3ca1510f01c68ab683 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
W20260709 13:00:34.100270  7561 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:34.101589  7747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f891b1555d4bdf94235c996aef9354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.103474  7673 tablet_service.cc:1511] Processing CreateTablet for tablet 28f891b1555d4bdf94235c996aef9354 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:00:34.107718  7674 tablet_service.cc:1511] Processing CreateTablet for tablet 1d44fab36da747c08f6c6b34bba33782 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:00:34.094750  7675 tablet_service.cc:1511] Processing CreateTablet for tablet 968f7bdc989246688fc4e10e8fd870a2 (DEFAULT_TABLE table=test-workload [id=635d3f0b5ea94cab99e5e4ca883b5f92]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:00:34.109948  7678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e834a2c2ef24568a8505066a5b8bdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.110832  7749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 968f7bdc989246688fc4e10e8fd870a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.114787  7676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d74af98f944e3ca1510f01c68ab683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.117581  7674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d44fab36da747c08f6c6b34bba33782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.118196  7750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d74af98f944e3ca1510f01c68ab683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.118467  7673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f891b1555d4bdf94235c996aef9354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.121183  7675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 968f7bdc989246688fc4e10e8fd870a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.119653  7754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b6ec9f23a94abea53c55ef57a0fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.124682  7751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4cb96a3fc044fd92d7be13e0807e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.128461  7680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b6ec9f23a94abea53c55ef57a0fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.137905  7677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4cb96a3fc044fd92d7be13e0807e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.135408  7748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d44fab36da747c08f6c6b34bba33782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:34.204414  7807 tablet_bootstrap.cc:492] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:34.204931  7807 ts_tablet_manager.cc:1403] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.200s	user 0.021s	sys 0.015s
I20260709 13:00:34.244159  7807 raft_consensus.cc:359] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.244993  7807 raft_consensus.cc:385] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.245317  7807 raft_consensus.cc:740] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.268707  7810 tablet_bootstrap.cc:492] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
I20260709 13:00:34.271056  7807 consensus_queue.cc:260] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.276752  7809 tablet_bootstrap.cc:492] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.282634  7641 heartbeater.cc:499] Master 127.7.78.126:39787 was elected leader, sending a full tablet report...
I20260709 13:00:34.282922  7807 ts_tablet_manager.cc:1434] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.078s	user 0.009s	sys 0.004s
I20260709 13:00:34.283900  7807 tablet_bootstrap.cc:492] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:34.299149  7809 tablet_bootstrap.cc:654] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.304723  7810 tablet_bootstrap.cc:654] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.314316  7807 tablet_bootstrap.cc:654] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.315188  7809 tablet_bootstrap.cc:492] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.315763  7809 ts_tablet_manager.cc:1403] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.053s	user 0.018s	sys 0.000s
I20260709 13:00:34.318722  7810 tablet_bootstrap.cc:492] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.319284  7810 ts_tablet_manager.cc:1403] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.053s	user 0.019s	sys 0.003s
I20260709 13:00:34.319831  7809 raft_consensus.cc:359] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.320698  7809 raft_consensus.cc:385] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.321044  7809 raft_consensus.cc:740] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.322294  7809 consensus_queue.cc:260] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.322600  7810 raft_consensus.cc:359] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.323472  7810 raft_consensus.cc:385] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.323830  7810 raft_consensus.cc:740] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.324647  7810 consensus_queue.cc:260] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.330150  7809 ts_tablet_manager.cc:1434] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.011s
I20260709 13:00:34.330973  7790 heartbeater.cc:499] Master 127.7.78.126:39787 was elected leader, sending a full tablet report...
I20260709 13:00:34.331226  7809 tablet_bootstrap.cc:492] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.333300  7716 heartbeater.cc:499] Master 127.7.78.126:39787 was elected leader, sending a full tablet report...
I20260709 13:00:34.334404  7810 ts_tablet_manager.cc:1434] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.005s
I20260709 13:00:34.335875  7810 tablet_bootstrap.cc:492] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
I20260709 13:00:34.340394  7809 tablet_bootstrap.cc:654] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.342834  7810 tablet_bootstrap.cc:654] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.353227  7809 tablet_bootstrap.cc:492] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.353830  7809 ts_tablet_manager.cc:1403] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.008s
I20260709 13:00:34.354728  7810 tablet_bootstrap.cc:492] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.355216  7810 ts_tablet_manager.cc:1403] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.006s
I20260709 13:00:34.358513  7807 tablet_bootstrap.cc:492] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:34.357971  7809 raft_consensus.cc:359] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.358207  7810 raft_consensus.cc:359] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.358994  7809 raft_consensus.cc:385] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.359102  7807 ts_tablet_manager.cc:1403] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.075s	user 0.012s	sys 0.001s
I20260709 13:00:34.359122  7810 raft_consensus.cc:385] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.359448  7809 raft_consensus.cc:740] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.359681  7810 raft_consensus.cc:740] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
W20260709 13:00:34.359827  7642 tablet.cc:2406] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:00:34.360452  7809 consensus_queue.cc:260] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.360729  7810 consensus_queue.cc:260] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.363440  7809 ts_tablet_manager.cc:1434] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:34.363467  7810 ts_tablet_manager.cc:1434] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:00:34.364763  7810 tablet_bootstrap.cc:492] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
I20260709 13:00:34.364845  7809 tablet_bootstrap.cc:492] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.372478  7810 tablet_bootstrap.cc:654] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.373104  7809 tablet_bootstrap.cc:654] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.385013  7810 tablet_bootstrap.cc:492] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.385565  7810 ts_tablet_manager.cc:1403] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.021s	user 0.018s	sys 0.002s
I20260709 13:00:34.385860  7809 tablet_bootstrap.cc:492] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.386354  7809 ts_tablet_manager.cc:1403] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.022s	user 0.017s	sys 0.002s
I20260709 13:00:34.388692  7810 raft_consensus.cc:359] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.389467  7810 raft_consensus.cc:385] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.389793  7810 raft_consensus.cc:740] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.389575  7809 raft_consensus.cc:359] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.390408  7809 raft_consensus.cc:385] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.390776  7809 raft_consensus.cc:740] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.390609  7810 consensus_queue.cc:260] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.391635  7809 consensus_queue.cc:260] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.393584  7810 ts_tablet_manager.cc:1434] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:00:34.394546  7809 ts_tablet_manager.cc:1434] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:00:34.394737  7810 tablet_bootstrap.cc:492] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
I20260709 13:00:34.395679  7809 tablet_bootstrap.cc:492] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.402336  7810 tablet_bootstrap.cc:654] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.403789  7809 tablet_bootstrap.cc:654] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.412550  7811 raft_consensus.cc:493] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.413251  7811 raft_consensus.cc:515] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.415398  7810 tablet_bootstrap.cc:492] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.415920  7810 ts_tablet_manager.cc:1403] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.012s
I20260709 13:00:34.416828  7811 leader_election.cc:290] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.417380  7809 tablet_bootstrap.cc:492] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.417912  7809 ts_tablet_manager.cc:1403] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.022s	user 0.016s	sys 0.004s
I20260709 13:00:34.421622  7809 raft_consensus.cc:359] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.422876  7809 raft_consensus.cc:385] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.423193  7809 raft_consensus.cc:740] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.424052  7809 consensus_queue.cc:260] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.425763  7810 raft_consensus.cc:359] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.426548  7810 raft_consensus.cc:385] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.426868  7810 raft_consensus.cc:740] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.426983  7809 ts_tablet_manager.cc:1434] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:34.428081  7809 tablet_bootstrap.cc:492] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.433933  7810 consensus_queue.cc:260] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.434947  7809 tablet_bootstrap.cc:654] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.437038  7810 ts_tablet_manager.cc:1434] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 13:00:34.438063  7810 tablet_bootstrap.cc:492] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
I20260709 13:00:34.440780  7807 raft_consensus.cc:359] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.442092  7807 raft_consensus.cc:385] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.442580  7807 raft_consensus.cc:740] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.447894  7809 tablet_bootstrap.cc:492] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.448544  7809 ts_tablet_manager.cc:1403] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.004s
I20260709 13:00:34.452160  7809 raft_consensus.cc:359] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.453195  7809 raft_consensus.cc:385] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.453707  7809 raft_consensus.cc:740] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.453711  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:34.454697  7815 raft_consensus.cc:493] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.454720  7691 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 0.
I20260709 13:00:34.455328  7815 raft_consensus.cc:515] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.458390  7815 leader_election.cc:290] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:34.459026  7579 leader_election.cc:304] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:34.459208  7809 consensus_queue.cc:260] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.460119  7811 raft_consensus.cc:2804] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.460484  7811 raft_consensus.cc:493] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.460824  7811 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.462059  7809 ts_tablet_manager.cc:1434] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 13:00:34.463212  7809 tablet_bootstrap.cc:492] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.469039  7810 tablet_bootstrap.cc:654] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.444453  7807 consensus_queue.cc:260] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.484439  7815 raft_consensus.cc:493] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.468993  7811 raft_consensus.cc:515] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.489100  7815 raft_consensus.cc:515] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.491515  7815 leader_election.cc:290] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:34.469866  7809 tablet_bootstrap.cc:654] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.492635  7811 leader_election.cc:290] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.493875  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:34.494534  7691 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.500500  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:34.501016  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:34.502975  7807 ts_tablet_manager.cc:1434] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.143s	user 0.011s	sys 0.002s
W20260709 13:00:34.504287  7580 leader_election.cc:343] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:34.505438  7580 leader_election.cc:343] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.505697  7691 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 1.
I20260709 13:00:34.506862  7579 leader_election.cc:304] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 76ae0ee745734d5e9088ff6563d716d8
I20260709 13:00:34.507558  7807 tablet_bootstrap.cc:492] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:34.512395  7809 tablet_bootstrap.cc:492] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.512998  7809 ts_tablet_manager.cc:1403] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.050s	user 0.009s	sys 0.006s
I20260709 13:00:34.515144  7810 tablet_bootstrap.cc:492] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.515646  7810 ts_tablet_manager.cc:1403] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.078s	user 0.006s	sys 0.027s
I20260709 13:00:34.516294  7809 raft_consensus.cc:359] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.517030  7809 raft_consensus.cc:385] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.517350  7809 raft_consensus.cc:740] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.518052  7809 consensus_queue.cc:260] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.518574  7811 raft_consensus.cc:2804] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:34.520344  7811 raft_consensus.cc:697] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:34.521382  7811 consensus_queue.cc:237] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.523679  7816 raft_consensus.cc:493] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.524269  7816 raft_consensus.cc:515] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.518610  7810 raft_consensus.cc:359] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.527002  7816 leader_election.cc:290] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.527535  7810 raft_consensus.cc:385] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.528880  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.533704  7810 raft_consensus.cc:740] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.535355  7810 consensus_queue.cc:260] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.535941  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.538434  7810 ts_tablet_manager.cc:1434] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.002s
W20260709 13:00:34.538569  7729 leader_election.cc:343] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.539516  7810 tablet_bootstrap.cc:492] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
W20260709 13:00:34.539464  7729 leader_election.cc:343] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.543006  7807 tablet_bootstrap.cc:654] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.548422  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:34.549911  7690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:34.550784  7690 raft_consensus.cc:2468] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 0.
W20260709 13:00:34.550848  7728 leader_election.cc:343] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.551923  7728 leader_election.cc:304] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8; no voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:34.553740  7815 raft_consensus.cc:2749] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:34.555840  7809 ts_tablet_manager.cc:1434] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.000s
I20260709 13:00:34.555609  7728 leader_election.cc:304] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:34.556918  7809 tablet_bootstrap.cc:492] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.557256  7815 raft_consensus.cc:2804] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.557789  7815 raft_consensus.cc:493] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.558235  7815 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.572190  7815 raft_consensus.cc:515] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.577131  7810 tablet_bootstrap.cc:654] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.576510  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:34.580261  7815 leader_election.cc:290] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:34.581462  7690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:34.582085  7690 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:34.583889  7729 leader_election.cc:343] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.592475  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.590307  7513 catalog_manager.cc:5697] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 0 to 1, leader changed from <none> to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:34.594558  7807 tablet_bootstrap.cc:492] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:34.595117  7807 ts_tablet_manager.cc:1403] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.088s	user 0.020s	sys 0.008s
I20260709 13:00:34.603413  7690 raft_consensus.cc:2468] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 1.
I20260709 13:00:34.603286  7815 raft_consensus.cc:493] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.604964  7815 raft_consensus.cc:515] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.607718  7728 leader_election.cc:304] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:34.608042  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.609917  7807 raft_consensus.cc:359] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.613823  7807 raft_consensus.cc:385] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:00:34.613775  7729 leader_election.cc:343] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.613986  7815 leader_election.cc:290] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:34.618779  7690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
W20260709 13:00:34.620455  7728 leader_election.cc:343] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.620972  7728 leader_election.cc:304] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8; no voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:34.621619  7815 raft_consensus.cc:2804] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:34.614490  7807 raft_consensus.cc:740] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.625950  7838 raft_consensus.cc:2749] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:34.627848  7655 leader_election.cc:343] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.629911  7807 consensus_queue.cc:260] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.638909  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:34.639791  7807 ts_tablet_manager.cc:1434] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.044s	user 0.005s	sys 0.003s
I20260709 13:00:34.640143  7810 tablet_bootstrap.cc:492] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.640646  7810 ts_tablet_manager.cc:1403] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.101s	user 0.011s	sys 0.017s
I20260709 13:00:34.640753  7807 tablet_bootstrap.cc:492] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
W20260709 13:00:34.643339  7655 leader_election.cc:343] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.643906  7810 raft_consensus.cc:359] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.645673  7810 raft_consensus.cc:385] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.645993  7810 raft_consensus.cc:740] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.643963  7655 leader_election.cc:304] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b523fc673cf34021a277e15cd85c3b20; no voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:34.646196  7809 tablet_bootstrap.cc:654] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.647759  7810 consensus_queue.cc:260] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.650687  7816 raft_consensus.cc:2749] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:34.652729  7810 ts_tablet_manager.cc:1434] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.004s
I20260709 13:00:34.653934  7838 raft_consensus.cc:493] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.654736  7838 raft_consensus.cc:515] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.655401  7815 raft_consensus.cc:697] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Becoming Leader. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:34.657442  7839 raft_consensus.cc:493] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.658125  7839 raft_consensus.cc:515] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.658977  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.660763  7807 tablet_bootstrap.cc:654] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.657220  7838 leader_election.cc:290] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
W20260709 13:00:34.660641  7729 leader_election.cc:343] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.661098  7839 leader_election.cc:290] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:34.656463  7815 consensus_queue.cc:237] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.663232  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.663914  7616 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 0.
I20260709 13:00:34.665645  7729 leader_election.cc:304] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:34.666509  7839 raft_consensus.cc:2804] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.668640  7690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:34.669183  7810 tablet_bootstrap.cc:492] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
W20260709 13:00:34.670269  7728 leader_election.cc:343] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.670390  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:34.670771  7728 leader_election.cc:304] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8; no voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:34.671038  7691 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 0.
I20260709 13:00:34.671566  7838 raft_consensus.cc:2749] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:34.667024  7839 raft_consensus.cc:493] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.672331  7839 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.677964  7810 tablet_bootstrap.cc:654] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.689759  7839 raft_consensus.cc:515] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.693903  7839 leader_election.cc:290] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:34.696534  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:34.697414  7691 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.715313  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:34.715976  7616 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.717024  7809 tablet_bootstrap.cc:492] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.717538  7809 ts_tablet_manager.cc:1403] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.161s	user 0.010s	sys 0.007s
I20260709 13:00:34.719933  7809 raft_consensus.cc:359] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.720499  7809 raft_consensus.cc:385] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.720762  7809 raft_consensus.cc:740] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.723357  7616 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 1.
I20260709 13:00:34.723534  7513 catalog_manager.cc:5697] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "76ae0ee745734d5e9088ff6563d716d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:34.724632  7729 leader_election.cc:304] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:34.725687  7815 raft_consensus.cc:2804] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:34.726428  7815 raft_consensus.cc:697] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Becoming Leader. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:34.727313  7815 consensus_queue.cc:237] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.730603  7841 raft_consensus.cc:493] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.731112  7841 raft_consensus.cc:515] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.734455  7807 tablet_bootstrap.cc:492] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:34.734939  7807 ts_tablet_manager.cc:1403] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.094s	user 0.015s	sys 0.014s
I20260709 13:00:34.735736  7691 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 1.
I20260709 13:00:34.737028  7841 leader_election.cc:290] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.738561  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:34.739315  7764 raft_consensus.cc:2468] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 0.
I20260709 13:00:34.739461  7807 raft_consensus.cc:359] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.740243  7807 raft_consensus.cc:385] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.740592  7807 raft_consensus.cc:740] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.740922  7655 leader_election.cc:304] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:34.741346  7807 consensus_queue.cc:260] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.741977  7841 raft_consensus.cc:2804] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.742368  7841 raft_consensus.cc:493] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.742724  7841 raft_consensus.cc:3060] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.721349  7809 consensus_queue.cc:260] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.746722  7809 ts_tablet_manager.cc:1434] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260709 13:00:34.747772  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.748463  7616 raft_consensus.cc:2468] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 0.
I20260709 13:00:34.749819  7809 tablet_bootstrap.cc:492] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.751152  7841 raft_consensus.cc:515] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.755306  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:34.755065  7513 catalog_manager.cc:5697] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "76ae0ee745734d5e9088ff6563d716d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:34.755918  7764 raft_consensus.cc:3060] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.756865  7809 tablet_bootstrap.cc:654] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.759315  7807 ts_tablet_manager.cc:1434] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 13:00:34.766961  7807 tablet_bootstrap.cc:492] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:34.773584  7807 tablet_bootstrap.cc:654] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.774026  7841 leader_election.cc:290] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.775295  7764 raft_consensus.cc:2468] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 1.
I20260709 13:00:34.778662  7655 leader_election.cc:304] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:34.779989  7841 raft_consensus.cc:2804] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:34.776243  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:34.782172  7616 raft_consensus.cc:3060] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.780845  7841 raft_consensus.cc:697] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:34.786320  7841 consensus_queue.cc:237] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.788774  7810 tablet_bootstrap.cc:492] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.798352  7810 ts_tablet_manager.cc:1403] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.129s	user 0.012s	sys 0.017s
I20260709 13:00:34.804718  7810 raft_consensus.cc:359] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.805505  7810 raft_consensus.cc:385] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.806097  7810 raft_consensus.cc:740] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.806846  7810 consensus_queue.cc:260] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.807595  7616 raft_consensus.cc:2468] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 1.
I20260709 13:00:34.809563  7810 ts_tablet_manager.cc:1434] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:00:34.810652  7810 tablet_bootstrap.cc:492] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
I20260709 13:00:34.811224  7809 tablet_bootstrap.cc:492] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.811786  7809 ts_tablet_manager.cc:1403] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.062s	user 0.033s	sys 0.026s
I20260709 13:00:34.794092  7807 tablet_bootstrap.cc:492] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:34.813318  7807 ts_tablet_manager.cc:1403] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.047s	user 0.012s	sys 0.000s
I20260709 13:00:34.814642  7809 raft_consensus.cc:359] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.815358  7809 raft_consensus.cc:385] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.815686  7809 raft_consensus.cc:740] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.816267  7807 raft_consensus.cc:359] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.816493  7809 consensus_queue.cc:260] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.817022  7807 raft_consensus.cc:385] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.817051  7512 catalog_manager.cc:5697] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:34.818886  7809 ts_tablet_manager.cc:1434] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:00:34.819852  7809 tablet_bootstrap.cc:492] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8: Bootstrap starting.
I20260709 13:00:34.828261  7809 tablet_bootstrap.cc:654] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.817559  7807 raft_consensus.cc:740] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.836407  7810 tablet_bootstrap.cc:654] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.837229  7807 consensus_queue.cc:260] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.844063  7811 raft_consensus.cc:493] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.844626  7811 raft_consensus.cc:515] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.841385  7807 ts_tablet_manager.cc:1434] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.001s
I20260709 13:00:34.846131  7841 raft_consensus.cc:493] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.846766  7811 leader_election.cc:290] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.846911  7809 tablet_bootstrap.cc:492] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8: No bootstrap required, opened a new log
I20260709 13:00:34.846592  7841 raft_consensus.cc:515] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.847386  7809 ts_tablet_manager.cc:1403] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.008s
I20260709 13:00:34.848009  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:34.848544  7841 leader_election.cc:290] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.848616  7691 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 0.
I20260709 13:00:34.849892  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:34.850307  7579 leader_election.cc:304] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:34.852088  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.852710  7616 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 0.
I20260709 13:00:34.850510  7764 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 0.
I20260709 13:00:34.853685  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:34.854265  7811 raft_consensus.cc:2804] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.854772  7811 raft_consensus.cc:493] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.855633  7811 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.855403  7809 raft_consensus.cc:359] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.855206  7841 raft_consensus.cc:493] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.856372  7807 tablet_bootstrap.cc:492] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:34.855429  7655 leader_election.cc:304] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:34.856541  7809 raft_consensus.cc:385] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.854256  7765 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 0.
I20260709 13:00:34.856931  7809 raft_consensus.cc:740] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.857028  7816 raft_consensus.cc:2804] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.857429  7816 raft_consensus.cc:493] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.857750  7816 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.857690  7809 consensus_queue.cc:260] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.858940  7841 raft_consensus.cc:515] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.859817  7809 ts_tablet_manager.cc:1434] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:00:34.860097  7815 raft_consensus.cc:493] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.860597  7815 raft_consensus.cc:515] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.864128  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:34.864741  7815 leader_election.cc:290] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:34.865061  7691 raft_consensus.cc:2393] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76ae0ee745734d5e9088ff6563d716d8 in current term 1: Already voted for candidate a791c55d7d0a4d91a10e02f61f7a1662 in this term.
I20260709 13:00:34.865468  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.866582  7841 leader_election.cc:290] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.867143  7729 leader_election.cc:304] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8; no voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:34.868911  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.864377  7807 tablet_bootstrap.cc:654] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.868436  7815 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:34.870971  7655 leader_election.cc:343] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.872290  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:34.874296  7765 raft_consensus.cc:2468] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 0.
I20260709 13:00:34.877030  7810 tablet_bootstrap.cc:492] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.877550  7810 ts_tablet_manager.cc:1403] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.067s	user 0.016s	sys 0.005s
I20260709 13:00:34.877724  7655 leader_election.cc:304] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:34.880422  7841 raft_consensus.cc:2804] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.880823  7841 raft_consensus.cc:493] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.881170  7841 raft_consensus.cc:3060] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.881587  7816 raft_consensus.cc:515] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.887929  7811 raft_consensus.cc:515] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.889671  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:34.890568  7815 raft_consensus.cc:2749] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:34.894220  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:34.895195  7765 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.890725  7816 leader_election.cc:290] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.902675  7616 raft_consensus.cc:2393] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b523fc673cf34021a277e15cd85c3b20 in current term 1: Already voted for candidate a791c55d7d0a4d91a10e02f61f7a1662 in this term.
I20260709 13:00:34.908069  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:34.908272  7841 raft_consensus.cc:515] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.909001  7691 raft_consensus.cc:2393] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a791c55d7d0a4d91a10e02f61f7a1662 in current term 1: Already voted for candidate b523fc673cf34021a277e15cd85c3b20 in this term.
I20260709 13:00:34.910861  7841 leader_election.cc:290] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.911942  7811 leader_election.cc:290] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.912844  7810 raft_consensus.cc:359] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.913223  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:34.913893  7810 raft_consensus.cc:385] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.914076  7764 raft_consensus.cc:3060] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.914407  7810 raft_consensus.cc:740] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.915172  7810 consensus_queue.cc:260] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.915452  7763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:34.918116  7580 leader_election.cc:304] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662; no voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:34.913781  7615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
W20260709 13:00:34.920464  7655 leader_election.cc:343] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:34.917614  7810 ts_tablet_manager.cc:1434] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.040s	user 0.009s	sys 0.000s
I20260709 13:00:34.921866  7764 raft_consensus.cc:2468] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 1.
I20260709 13:00:34.923136  7655 leader_election.cc:304] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:34.924047  7810 tablet_bootstrap.cc:492] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20: Bootstrap starting.
I20260709 13:00:34.925971  7816 raft_consensus.cc:2804] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:34.925140  7765 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 1.
I20260709 13:00:34.925758  7811 raft_consensus.cc:2749] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:34.931895  7816 raft_consensus.cc:697] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:34.934296  7655 leader_election.cc:304] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:34.935065  7841 raft_consensus.cc:2804] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:34.936308  7841 raft_consensus.cc:697] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:34.936576  7807 tablet_bootstrap.cc:492] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:34.937088  7807 ts_tablet_manager.cc:1403] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.081s	user 0.022s	sys 0.028s
I20260709 13:00:34.938205  7810 tablet_bootstrap.cc:654] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.938047  7816 consensus_queue.cc:237] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.937234  7841 consensus_queue.cc:237] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.942685  7807 raft_consensus.cc:359] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.943797  7807 raft_consensus.cc:385] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.944286  7807 raft_consensus.cc:740] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.948781  7854 raft_consensus.cc:493] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:34.949487  7807 consensus_queue.cc:260] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.949354  7854 raft_consensus.cc:515] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.953086  7807 ts_tablet_manager.cc:1434] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 13:00:34.954836  7854 leader_election.cc:290] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.956900  7807 tablet_bootstrap.cc:492] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:34.958215  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" is_pre_election: true
I20260709 13:00:34.959228  7616 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 0.
I20260709 13:00:34.960912  7655 leader_election.cc:304] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:34.962107  7854 raft_consensus.cc:2804] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:34.962507  7854 raft_consensus.cc:493] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:34.962594  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:34.962935  7854 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.963331  7765 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 0.
I20260709 13:00:34.964576  7807 tablet_bootstrap.cc:654] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:34.970219  7854 raft_consensus.cc:515] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.971894  7810 tablet_bootstrap.cc:492] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20: No bootstrap required, opened a new log
I20260709 13:00:34.972390  7810 ts_tablet_manager.cc:1403] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20: Time spent bootstrapping tablet: real 0.049s	user 0.022s	sys 0.002s
I20260709 13:00:34.975346  7810 raft_consensus.cc:359] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.976110  7810 raft_consensus.cc:385] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:34.976398  7810 raft_consensus.cc:740] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Initialized, Role: FOLLOWER
I20260709 13:00:34.977246  7810 consensus_queue.cc:260] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:34.980121  7810 ts_tablet_manager.cc:1434] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:00:34.980715  7811 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:34.981701  7510 catalog_manager.cc:5697] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:34.984009  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:34.984774  7616 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.985055  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:34.985707  7764 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:34.987812  7510 catalog_manager.cc:5697] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:34.993185  7854 leader_election.cc:290] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:34.999178  7616 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 1.
I20260709 13:00:35.001086  7655 leader_election.cc:304] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:35.002559  7854 raft_consensus.cc:2804] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:35.003338  7854 raft_consensus.cc:697] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:35.004387  7854 consensus_queue.cc:237] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.005051  7764 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 1.
I20260709 13:00:35.007279  7807 tablet_bootstrap.cc:492] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:35.007864  7807 ts_tablet_manager.cc:1403] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.051s	user 0.015s	sys 0.009s
I20260709 13:00:35.012972  7807 raft_consensus.cc:359] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.014091  7807 raft_consensus.cc:385] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:35.014468  7807 raft_consensus.cc:740] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:35.015302  7807 consensus_queue.cc:260] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.020920  7807 ts_tablet_manager.cc:1434] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:00:35.022156  7807 tablet_bootstrap.cc:492] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:35.031836  7807 tablet_bootstrap.cc:654] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:35.031760  7512 catalog_manager.cc:5697] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.043922  7807 tablet_bootstrap.cc:492] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:35.044523  7807 ts_tablet_manager.cc:1403] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.023s	user 0.018s	sys 0.000s
I20260709 13:00:35.047456  7807 raft_consensus.cc:359] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.048393  7807 raft_consensus.cc:385] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:35.048723  7807 raft_consensus.cc:740] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:35.049459  7807 consensus_queue.cc:260] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.054157  7811 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:35.057678  7807 ts_tablet_manager.cc:1434] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:00:35.071382  7807 tablet_bootstrap.cc:492] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662: Bootstrap starting.
I20260709 13:00:35.081974  7807 tablet_bootstrap.cc:654] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:35.094434  7807 tablet_bootstrap.cc:492] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662: No bootstrap required, opened a new log
I20260709 13:00:35.095011  7807 ts_tablet_manager.cc:1403] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent bootstrapping tablet: real 0.024s	user 0.016s	sys 0.003s
I20260709 13:00:35.098066  7807 raft_consensus.cc:359] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.098897  7807 raft_consensus.cc:385] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:35.099346  7807 raft_consensus.cc:740] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Initialized, Role: FOLLOWER
I20260709 13:00:35.100242  7807 consensus_queue.cc:260] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.102576  7807 ts_tablet_manager.cc:1434] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
W20260709 13:00:35.117156  7561 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:35.123956  7862 raft_consensus.cc:493] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.124509  7862 raft_consensus.cc:515] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.126724  7862 leader_election.cc:290] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.127684  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:35.128052  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:35.128492  7691 raft_consensus.cc:2468] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 0.
I20260709 13:00:35.128731  7765 raft_consensus.cc:2468] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 0.
I20260709 13:00:35.129746  7579 leader_election.cc:304] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:35.130710  7862 raft_consensus.cc:2804] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:35.131064  7862 raft_consensus.cc:493] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:35.131384  7862 raft_consensus.cc:3060] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.138502  7815 consensus_queue.cc:1048] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:35.138520  7862 raft_consensus.cc:515] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.141531  7690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:35.141731  7862 leader_election.cc:290] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.142132  7690 raft_consensus.cc:3060] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.147302  7862 raft_consensus.cc:493] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.147827  7862 raft_consensus.cc:515] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.149880  7862 leader_election.cc:290] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.152201  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:35.152945  7764 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 0.
I20260709 13:00:35.153290  7616 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.155252  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:35.156049  7689 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 0.
I20260709 13:00:35.157222  7579 leader_election.cc:304] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:35.158409  7862 raft_consensus.cc:2804] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:35.158803  7862 raft_consensus.cc:493] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:35.159236  7862 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.161707  7815 consensus_queue.cc:1048] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.164659  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:35.165460  7765 raft_consensus.cc:3060] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.166280  7862 raft_consensus.cc:515] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.168936  7862 leader_election.cc:290] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.169528  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:35.170243  7689 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.174129  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:35.174796  7764 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.167557  7690 raft_consensus.cc:2468] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 1.
I20260709 13:00:35.178844  7579 leader_election.cc:304] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:35.182008  7862 raft_consensus.cc:2804] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:35.182569  7862 raft_consensus.cc:697] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:35.183439  7862 consensus_queue.cc:237] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.184471  7765 raft_consensus.cc:2468] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 1.
I20260709 13:00:35.191563  7689 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 1.
I20260709 13:00:35.193495  7579 leader_election.cc:304] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:35.195369  7764 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 1.
I20260709 13:00:35.196220  7862 raft_consensus.cc:2804] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:35.196885  7862 raft_consensus.cc:697] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:35.197741  7862 consensus_queue.cc:237] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.198205  7510 catalog_manager.cc:5697] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 0 to 1, leader changed from <none> to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.210166  7815 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.211221  7862 raft_consensus.cc:493] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:35.211711  7862 raft_consensus.cc:515] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.213213  7510 catalog_manager.cc:5697] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 0 to 1, leader changed from <none> to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.214815  7862 leader_election.cc:290] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.215108  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523fc673cf34021a277e15cd85c3b20" is_pre_election: true
I20260709 13:00:35.221493  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ae0ee745734d5e9088ff6563d716d8" is_pre_election: true
I20260709 13:00:35.222451  7764 raft_consensus.cc:2393] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a791c55d7d0a4d91a10e02f61f7a1662 in current term 1: Already voted for candidate b523fc673cf34021a277e15cd85c3b20 in this term.
I20260709 13:00:35.223757  7580 leader_election.cc:304] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662; no voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:35.224581  7862 raft_consensus.cc:3060] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:35.231647  7862 raft_consensus.cc:2749] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:35.234232  7815 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.311278  7854 consensus_queue.cc:1048] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.324862  7841 consensus_queue.cc:1048] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.372504  7841 consensus_queue.cc:1048] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.387284  7841 consensus_queue.cc:1048] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.421248  7689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f3128520a7c344758ca9dac16999e350"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:44566
I20260709 13:00:35.422057  7689 raft_consensus.cc:606] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Received request to transfer leadership to TS a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:35.432265  7841 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.440974  7841 consensus_queue.cc:1048] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.441807  7764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
mode: GRACEFUL
new_leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:53948
I20260709 13:00:35.443287  7764 raft_consensus.cc:606] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Received request to transfer leadership to TS a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:35.449440  7816 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.465724  7841 consensus_queue.cc:1048] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:35.471606  7765 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
mode: GRACEFUL
new_leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:53948
I20260709 13:00:35.472254  7765 raft_consensus.cc:606] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Received request to transfer leadership to TS a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:35.510850  7615 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
mode: GRACEFUL
new_leader_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:59728
I20260709 13:00:35.511489  7615 raft_consensus.cc:606] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Received request to transfer leadership to TS b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:35.523639  7615 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
mode: GRACEFUL
new_leader_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:59728
I20260709 13:00:35.524278  7615 raft_consensus.cc:606] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Received request to transfer leadership to TS b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:35.530263  7689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:44566
I20260709 13:00:35.530789  7689 raft_consensus.cc:606] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Received request to transfer leadership to TS 76ae0ee745734d5e9088ff6563d716d8
I20260709 13:00:35.589816  7815 raft_consensus.cc:993] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8: : Instructing follower a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:35.590338  7838 raft_consensus.cc:1081] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Signalling peer a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:35.592243  7615 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:39290
I20260709 13:00:35.592841  7615 raft_consensus.cc:493] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:35.593487  7615 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.602707  7615 raft_consensus.cc:515] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.604894  7615 leader_election.cc:290] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.605551  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:35.605937  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:35.606204  7689 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.606573  7764 raft_consensus.cc:3055] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:35.606933  7764 raft_consensus.cc:740] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:35.608060  7764 consensus_queue.cc:260] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.609638  7764 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.613994  7689 raft_consensus.cc:2468] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:35.615453  7579 leader_election.cc:304] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:35.616413  7887 raft_consensus.cc:2804] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:35.616883  7887 raft_consensus.cc:697] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:35.616988  7764 raft_consensus.cc:2468] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:35.617759  7887 consensus_queue.cc:237] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.624675  7513 catalog_manager.cc:5697] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 1 to 2, leader changed from 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67) to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.655990  7887 consensus_queue.cc:1048] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.663851  7811 raft_consensus.cc:993] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662: : Instructing follower b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:35.664306  7811 raft_consensus.cc:1081] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Signalling peer b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:35.666019  7691 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:56180
I20260709 13:00:35.666575  7691 raft_consensus.cc:493] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:35.666952  7691 raft_consensus.cc:3060] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.670270  7887 consensus_queue.cc:1048] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:35.674137  7691 raft_consensus.cc:515] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.676491  7689 raft_consensus.cc:1240] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Rejecting Update request from peer a791c55d7d0a4d91a10e02f61f7a1662 for earlier term 1. Current term is 2. Ops: []
I20260709 13:00:35.677057  7691 leader_election.cc:290] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 2 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.676930  7615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:35.677627  7615 raft_consensus.cc:3055] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:35.677944  7615 raft_consensus.cc:740] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:35.679009  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d44fab36da747c08f6c6b34bba33782" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:35.682950  7615 consensus_queue.cc:260] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.683789  7862 consensus_queue.cc:1059] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:00:35.685593  7887 consensus_queue.cc:1175] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "76ae0ee745734d5e9088ff6563d716d8" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260709 13:00:35.689621  7615 raft_consensus.cc:3060] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.697901  7615 raft_consensus.cc:2468] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 2.
I20260709 13:00:35.698966  7655 leader_election.cc:304] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 76ae0ee745734d5e9088ff6563d716d8
I20260709 13:00:35.699798  7841 raft_consensus.cc:2804] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:35.700260  7841 raft_consensus.cc:697] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 2 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:35.700959  7841 consensus_queue.cc:237] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.707553  7513 catalog_manager.cc:5697] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 1 to 2, leader changed from a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65) to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.737538  7815 raft_consensus.cc:993] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8: : Instructing follower a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:35.738061  7838 raft_consensus.cc:1081] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Signalling peer a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:35.739850  7615 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:39290
I20260709 13:00:35.740516  7615 raft_consensus.cc:493] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:35.740912  7615 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.747768  7615 raft_consensus.cc:515] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.749907  7615 leader_election.cc:290] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.750857  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:35.751497  7689 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.751418  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:35.752154  7765 raft_consensus.cc:3055] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:35.752496  7765 raft_consensus.cc:740] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:35.753041  7887 consensus_queue.cc:1048] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.753280  7765 consensus_queue.cc:260] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.754511  7765 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.761310  7765 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:35.762836  7580 leader_election.cc:304] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:35.763609  7689 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:35.763881  7887 raft_consensus.cc:2804] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:35.764400  7887 raft_consensus.cc:697] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:35.765259  7887 consensus_queue.cc:237] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.780272  7811 consensus_queue.cc:1048] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:35.780658  7513 catalog_manager.cc:5697] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 1 to 2, leader changed from 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67) to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.792948  7811 raft_consensus.cc:993] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662: : Instructing follower b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:35.793452  7811 raft_consensus.cc:1081] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Signalling peer b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:35.795935  7689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:56180
I20260709 13:00:35.799404  7689 raft_consensus.cc:493] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:35.802285  7689 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.808849  7689 raft_consensus.cc:515] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.810642  7689 leader_election.cc:290] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 2 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.811568  7615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:35.812163  7615 raft_consensus.cc:3055] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:35.812438  7615 raft_consensus.cc:740] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:35.812155  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:35.812901  7764 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.813143  7615 consensus_queue.cc:260] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.814250  7615 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.825294  7615 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 2.
I20260709 13:00:35.826503  7764 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 2.
I20260709 13:00:35.826742  7655 leader_election.cc:304] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:35.827608  7816 raft_consensus.cc:2804] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:35.828090  7816 raft_consensus.cc:697] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:35.828851  7816 consensus_queue.cc:237] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.833956  7877 raft_consensus.cc:993] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:35.834390  7877 raft_consensus.cc:1081] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Signalling peer a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:35.836123  7615 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f3128520a7c344758ca9dac16999e350"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:59722
I20260709 13:00:35.836704  7615 raft_consensus.cc:493] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:35.836436  7513 catalog_manager.cc:5697] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 1 to 2, leader changed from a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65) to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.837090  7615 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.843719  7615 raft_consensus.cc:515] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.845813  7615 leader_election.cc:290] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:35.846928  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:35.847429  7764 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.849898  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:35.850616  7689 raft_consensus.cc:3055] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:35.851104  7689 raft_consensus.cc:740] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:35.851986  7689 consensus_queue.cc:260] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.853221  7689 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.853370  7764 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:35.854650  7580 leader_election.cc:304] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:35.855502  7887 raft_consensus.cc:2804] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:35.855937  7887 raft_consensus.cc:697] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:35.856724  7887 consensus_queue.cc:237] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.860608  7689 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:35.866699  7513 catalog_manager.cc:5697] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 1 to 2, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:35.983108  7854 raft_consensus.cc:993] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:35.983636  7877 raft_consensus.cc:1081] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Signalling peer 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:35.985435  7764 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:53942
I20260709 13:00:35.986042  7764 raft_consensus.cc:493] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:35.986416  7764 raft_consensus.cc:3060] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.992988  7764 raft_consensus.cc:515] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:35.995074  7764 leader_election.cc:290] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 2 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:35.996301  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:35.996301  7615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28f891b1555d4bdf94235c996aef9354" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:35.997184  7615 raft_consensus.cc:3060] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:35.997175  7689 raft_consensus.cc:3055] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:35.997845  7689 raft_consensus.cc:740] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:35.999778  7689 consensus_queue.cc:260] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:36.001258  7689 raft_consensus.cc:3060] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:36.005666  7615 raft_consensus.cc:2468] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 2.
I20260709 13:00:36.007016  7729 leader_election.cc:304] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:36.007956  7838 raft_consensus.cc:2804] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:36.008430  7838 raft_consensus.cc:697] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Becoming Leader. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:36.009238  7838 consensus_queue.cc:237] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:36.010021  7689 raft_consensus.cc:2468] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 2.
I20260709 13:00:36.016884  7513 catalog_manager.cc:5697] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 reported cstate change: term changed from 1 to 2, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "76ae0ee745734d5e9088ff6563d716d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:36.060328  7764 raft_consensus.cc:1275] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.062197  7811 consensus_queue.cc:1048] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.074169  7689 raft_consensus.cc:1275] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.077750  7811 consensus_queue.cc:1048] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.221979  7764 raft_consensus.cc:3060] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:36.227026  7764 raft_consensus.cc:1275] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.228302  7877 consensus_queue.cc:1048] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.238308  7615 raft_consensus.cc:1275] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.239815  7765 raft_consensus.cc:1275] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.241349  7877 consensus_queue.cc:1048] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.241969  7887 consensus_queue.cc:1048] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.245132  7689 raft_consensus.cc:1275] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.246851  7887 consensus_queue.cc:1048] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.282902  7615 raft_consensus.cc:1275] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.284121  7816 consensus_queue.cc:1048] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.291786  7765 raft_consensus.cc:1275] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.294056  7841 consensus_queue.cc:1048] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.302076  7691 raft_consensus.cc:1275] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.303412  7862 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.312631  7764 raft_consensus.cc:1275] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.314252  7862 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.402992  7615 raft_consensus.cc:1275] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.404031  7838 consensus_queue.cc:1048] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:36.410847  7691 raft_consensus.cc:1275] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:36.412508  7838 consensus_queue.cc:1048] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:38.564291  7481 auto_leader_rebalancer-test.cc:316] The leader distribution is 
I20260709 13:00:38.565187  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:38.589526  7691 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:44574
I20260709 13:00:38.590207  7691 raft_consensus.cc:606] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Received request to transfer leadership to TS 76ae0ee745734d5e9088ff6563d716d8
I20260709 13:00:38.602833  7615 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
mode: GRACEFUL
new_leader_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:59742
I20260709 13:00:38.603410  7615 raft_consensus.cc:606] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Received request to transfer leadership to TS 76ae0ee745734d5e9088ff6563d716d8
I20260709 13:00:38.604696  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:00:38.605216  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:38.865834  7902 raft_consensus.cc:993] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:38.866305  7877 raft_consensus.cc:1081] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Signalling peer 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:38.867966  7764 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:53942
I20260709 13:00:38.868558  7764 raft_consensus.cc:493] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:38.868939  7764 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:38.875105  7764 raft_consensus.cc:515] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:38.876602  7764 leader_election.cc:290] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 2 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:38.877586  7615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:38.877736  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:38.878222  7615 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:38.878283  7689 raft_consensus.cc:3055] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:38.878674  7689 raft_consensus.cc:740] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:38.879356  7689 consensus_queue.cc:260] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:38.880275  7689 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:38.883188  7615 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 2.
I20260709 13:00:38.884377  7729 leader_election.cc:304] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:38.885106  7815 raft_consensus.cc:2804] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:38.885097  7689 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 2.
I20260709 13:00:38.886669  7815 raft_consensus.cc:697] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Becoming Leader. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:38.887470  7815 consensus_queue.cc:237] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:38.894508  7513 catalog_manager.cc:5697] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 reported cstate change: term changed from 1 to 2, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "76ae0ee745734d5e9088ff6563d716d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:38.963768  7895 raft_consensus.cc:993] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: : Instructing follower 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:38.964219  7898 raft_consensus.cc:1081] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Signalling peer 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:38.965801  7764 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:50576
I20260709 13:00:38.966337  7764 raft_consensus.cc:493] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:38.966636  7764 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:38.970727  7764 raft_consensus.cc:515] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:38.972180  7764 leader_election.cc:290] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 2 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:38.973143  7615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:38.973315  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:38.973769  7615 raft_consensus.cc:3055] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:38.973922  7689 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:38.974205  7615 raft_consensus.cc:740] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:38.974951  7615 consensus_queue.cc:260] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:38.975858  7615 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:38.978787  7689 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 2.
I20260709 13:00:38.979883  7728 leader_election.cc:304] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:38.980583  7815 raft_consensus.cc:2804] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:38.981093  7815 raft_consensus.cc:697] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Becoming Leader. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:38.981742  7615 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 2.
I20260709 13:00:38.981938  7815 consensus_queue.cc:237] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:38.989283  7513 catalog_manager.cc:5697] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 reported cstate change: term changed from 1 to 2, leader changed from a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65) to 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "76ae0ee745734d5e9088ff6563d716d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:39.309935  7615 raft_consensus.cc:1275] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:39.311182  7815 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:39.322054  7689 raft_consensus.cc:1275] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:39.326524  7912 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:39.368314  7615 raft_consensus.cc:1275] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:39.369807  7912 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:39.378877  7689 raft_consensus.cc:1275] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:39.380332  7815 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:39.606209  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:39.618623  7615 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f3128520a7c344758ca9dac16999e350"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
mode: GRACEFUL
new_leader_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:59728
I20260709 13:00:39.619221  7615 raft_consensus.cc:606] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Received request to transfer leadership to TS b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:39.624800  7615 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
mode: GRACEFUL
new_leader_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:59728
I20260709 13:00:39.625344  7615 raft_consensus.cc:606] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Received request to transfer leadership to TS b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:39.630812  7764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
mode: GRACEFUL
new_leader_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:53948
I20260709 13:00:39.631273  7764 raft_consensus.cc:606] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Received request to transfer leadership to TS b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:39.636154  7615 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
mode: GRACEFUL
new_leader_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:59728
I20260709 13:00:39.636714  7615 raft_consensus.cc:606] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Received request to transfer leadership to TS b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:39.647356  7764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
mode: GRACEFUL
new_leader_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:53948
I20260709 13:00:39.647976  7764 raft_consensus.cc:606] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Received request to transfer leadership to TS b523fc673cf34021a277e15cd85c3b20
I20260709 13:00:39.827862  7918 raft_consensus.cc:993] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662: : Instructing follower b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.828526  7898 raft_consensus.cc:1081] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Signalling peer b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.830070  7689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:56180
I20260709 13:00:39.830679  7689 raft_consensus.cc:493] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:39.831049  7689 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.835525  7689 raft_consensus.cc:515] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.837112  7689 leader_election.cc:290] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:39.838038  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:39.838310  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e834a2c2ef24568a8505066a5b8bdd3" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:39.838560  7616 raft_consensus.cc:3055] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:39.838829  7764 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.838883  7616 raft_consensus.cc:740] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:39.839676  7616 consensus_queue.cc:260] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.840549  7616 raft_consensus.cc:3060] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.843108  7764 raft_consensus.cc:2468] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:39.844302  7655 leader_election.cc:304] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:39.845073  7877 raft_consensus.cc:2804] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:39.845507  7616 raft_consensus.cc:2468] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:39.845634  7877 raft_consensus.cc:697] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:39.846498  7877 consensus_queue.cc:237] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.852913  7513 catalog_manager.cc:5697] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 2 to 3, leader changed from a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65) to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:39.856673  7918 raft_consensus.cc:993] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662: : Instructing follower b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.857175  7898 raft_consensus.cc:1081] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Signalling peer b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.858592  7689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f3128520a7c344758ca9dac16999e350"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:56180
I20260709 13:00:39.859067  7689 raft_consensus.cc:493] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:39.859359  7689 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.865561  7689 raft_consensus.cc:515] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.867627  7689 leader_election.cc:290] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:39.868458  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:39.868662  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3128520a7c344758ca9dac16999e350" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:39.868978  7616 raft_consensus.cc:3055] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:39.869275  7616 raft_consensus.cc:740] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:39.869311  7764 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.870162  7616 consensus_queue.cc:260] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.871045  7616 raft_consensus.cc:3060] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.873934  7764 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:39.874986  7655 leader_election.cc:304] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:39.875717  7616 raft_consensus.cc:2468] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:39.875765  7877 raft_consensus.cc:2804] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:39.876410  7877 raft_consensus.cc:697] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:39.877853  7877 consensus_queue.cc:237] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.884589  7513 catalog_manager.cc:5697] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 2 to 3, leader changed from a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65) to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:39.920421  7925 raft_consensus.cc:993] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8: : Instructing follower b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.920928  7926 raft_consensus.cc:1081] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Signalling peer b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.922792  7689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:56186
I20260709 13:00:39.923316  7689 raft_consensus.cc:493] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:39.923732  7689 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.928315  7689 raft_consensus.cc:515] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.929894  7689 leader_election.cc:290] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:39.930713  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:39.930943  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:39.931409  7616 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.931501  7764 raft_consensus.cc:3055] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:39.931865  7764 raft_consensus.cc:740] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:39.932600  7764 consensus_queue.cc:260] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.933764  7764 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.936604  7616 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:39.937817  7655 leader_election.cc:304] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:39.937994  7764 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:39.938616  7877 raft_consensus.cc:2804] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:39.939121  7877 raft_consensus.cc:697] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:39.939823  7877 consensus_queue.cc:237] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.945791  7513 catalog_manager.cc:5697] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 2 to 3, leader changed from 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67) to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:39.988677  7925 raft_consensus.cc:993] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8: : Instructing follower b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.989223  7926 raft_consensus.cc:1081] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Signalling peer b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:39.990957  7689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:56186
I20260709 13:00:39.991518  7689 raft_consensus.cc:493] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:39.991802  7689 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:39.997058  7689 raft_consensus.cc:515] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:39.998656  7689 leader_election.cc:290] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:39.999511  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:39.999737  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:40.000176  7616 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:40.000279  7764 raft_consensus.cc:3055] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:40.000625  7764 raft_consensus.cc:740] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:40.001271  7764 consensus_queue.cc:260] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:40.002169  7764 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:40.005861  7616 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:40.006186  7764 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:40.007200  7655 leader_election.cc:304] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a791c55d7d0a4d91a10e02f61f7a1662, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:40.007915  7877 raft_consensus.cc:2804] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:40.008378  7877 raft_consensus.cc:697] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:40.009027  7877 consensus_queue.cc:237] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:40.015116  7513 catalog_manager.cc:5697] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 2 to 3, leader changed from 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67) to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:40.142959  7918 raft_consensus.cc:993] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662: : Instructing follower b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:40.143425  7898 raft_consensus.cc:1081] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Signalling peer b523fc673cf34021a277e15cd85c3b20 to start an election
I20260709 13:00:40.145030  7689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
 from {username='slave'} at 127.0.0.1:56180
I20260709 13:00:40.145538  7689 raft_consensus.cc:493] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:40.145846  7689 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:40.149936  7689 raft_consensus.cc:515] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:40.151393  7689 leader_election.cc:290] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:40.152686  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:40.152705  7764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "b523fc673cf34021a277e15cd85c3b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:40.153317  7616 raft_consensus.cc:3055] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:40.153565  7764 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:40.153640  7616 raft_consensus.cc:740] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:40.154490  7616 consensus_queue.cc:260] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:40.155424  7616 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:40.158406  7764 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:40.159468  7655 leader_election.cc:304] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, b523fc673cf34021a277e15cd85c3b20; no voters: 
I20260709 13:00:40.160162  7877 raft_consensus.cc:2804] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:40.160403  7616 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b523fc673cf34021a277e15cd85c3b20 in term 3.
I20260709 13:00:40.160683  7877 raft_consensus.cc:697] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Leader. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:40.161540  7877 consensus_queue.cc:237] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:40.168344  7513 catalog_manager.cc:5697] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 reported cstate change: term changed from 2 to 3, leader changed from a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65) to b523fc673cf34021a277e15cd85c3b20 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "b523fc673cf34021a277e15cd85c3b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:40.289805  7616 raft_consensus.cc:1275] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.291203  7877 consensus_queue.cc:1048] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.301052  7764 raft_consensus.cc:1275] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.304152  7877 consensus_queue.cc:1048] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.352510  7616 raft_consensus.cc:1275] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.353749  7877 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.361690  7764 raft_consensus.cc:1275] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.363571  7919 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.392016  7616 raft_consensus.cc:1275] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.393460  7919 consensus_queue.cc:1048] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.402879  7764 raft_consensus.cc:1275] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.404400  7919 consensus_queue.cc:1048] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.462144  7616 raft_consensus.cc:1275] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.463384  7919 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.473724  7764 raft_consensus.cc:1275] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.475505  7937 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.646998  7764 raft_consensus.cc:1275] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.648191  7937 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:40.656926  7616 raft_consensus.cc:1275] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Refusing update from remote peer b523fc673cf34021a277e15cd85c3b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:40.659420  7919 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:42.656102  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:42.663727  7689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:44574
I20260709 13:00:42.664326  7689 raft_consensus.cc:606] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 LEADER]: Received request to transfer leadership to TS a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:42.667021  7689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:44574
I20260709 13:00:42.667392  7689 raft_consensus.cc:606] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Received request to transfer leadership to TS 76ae0ee745734d5e9088ff6563d716d8
I20260709 13:00:42.669548  7689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:44574
I20260709 13:00:42.669975  7689 raft_consensus.cc:606] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Received request to transfer leadership to TS 76ae0ee745734d5e9088ff6563d716d8
I20260709 13:00:42.672226  7689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:44574
I20260709 13:00:42.672708  7689 raft_consensus.cc:606] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Received request to transfer leadership to TS a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:42.675182  7689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429"
dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
mode: GRACEFUL
new_leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:44574
I20260709 13:00:42.675694  7689 raft_consensus.cc:606] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Received request to transfer leadership to TS a791c55d7d0a4d91a10e02f61f7a1662
I20260709 13:00:42.676898  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:00:42.677327  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:42.742043  7877 raft_consensus.cc:993] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:42.742468  7877 raft_consensus.cc:1081] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 LEADER]: Signalling peer a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:42.743964  7615 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:59722
I20260709 13:00:42.744431  7615 raft_consensus.cc:493] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:42.744730  7615 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:42.749584  7615 raft_consensus.cc:515] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:42.751330  7615 leader_election.cc:290] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 3 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:42.752241  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:42.752734  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f4cb96a3fc044fd92d7be13e0807e60" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:42.752923  7689 raft_consensus.cc:3055] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:42.753193  7689 raft_consensus.cc:740] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:42.753227  7765 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:42.754041  7689 consensus_queue.cc:260] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:42.755048  7689 raft_consensus.cc:3060] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:42.758497  7765 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 3.
I20260709 13:00:42.759757  7580 leader_election.cc:304] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:42.759882  7689 raft_consensus.cc:2468] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 3.
I20260709 13:00:42.761437  7946 raft_consensus.cc:2804] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:42.762243  7946 raft_consensus.cc:697] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:42.763130  7946 consensus_queue.cc:237] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:42.769647  7513 catalog_manager.cc:5697] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 2 to 3, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:42.974906  7945 raft_consensus.cc:993] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:42.975414  7877 raft_consensus.cc:1081] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Signalling peer 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:42.977131  7765 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:53942
I20260709 13:00:42.977909  7765 raft_consensus.cc:493] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:42.978318  7765 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:42.980983  7945 raft_consensus.cc:993] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:42.981571  7877 raft_consensus.cc:1081] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Signalling peer a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:42.983232  7615 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:59722
I20260709 13:00:42.983774  7615 raft_consensus.cc:493] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:42.984148  7615 raft_consensus.cc:3060] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:42.983803  7765 raft_consensus.cc:515] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:42.986217  7765 leader_election.cc:290] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 4 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:42.986800  7689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:42.986842  7616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "968f7bdc989246688fc4e10e8fd870a2" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:42.987582  7616 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:42.987514  7689 raft_consensus.cc:3055] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:42.987979  7689 raft_consensus.cc:740] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:42.988755  7689 consensus_queue.cc:260] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:42.990005  7689 raft_consensus.cc:3060] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:42.991355  7615 raft_consensus.cc:515] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:42.992268  7616 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 4.
I20260709 13:00:42.993682  7729 leader_election.cc:304] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:42.993877  7615 leader_election.cc:290] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:42.994524  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:42.994843  7926 raft_consensus.cc:2804] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:42.995112  7765 raft_consensus.cc:3060] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:42.996446  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d74af98f944e3ca1510f01c68ab683" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:42.997095  7691 raft_consensus.cc:3055] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:42.997519  7691 raft_consensus.cc:740] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:42.997269  7689 raft_consensus.cc:2468] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 4.
I20260709 13:00:42.998343  7691 consensus_queue.cc:260] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:42.999828  7691 raft_consensus.cc:3060] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:43.005545  7926 raft_consensus.cc:697] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 LEADER]: Becoming Leader. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:43.005508  7765 raft_consensus.cc:2468] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:43.006673  7926 consensus_queue.cc:237] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.007256  7691 raft_consensus.cc:2468] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 2.
I20260709 13:00:43.007527  7580 leader_election.cc:304] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:43.008952  7946 raft_consensus.cc:2804] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:43.009385  7946 raft_consensus.cc:697] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:43.010299  7946 consensus_queue.cc:237] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.016368  7513 catalog_manager.cc:5697] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 reported cstate change: term changed from 3 to 4, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67). New cstate: current_term: 4 leader_uuid: "76ae0ee745734d5e9088ff6563d716d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:43.018505  7512 catalog_manager.cc:5697] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 1 to 2, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:43.073989  7945 raft_consensus.cc:993] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:43.074499  7877 raft_consensus.cc:1081] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Signalling peer a791c55d7d0a4d91a10e02f61f7a1662 to start an election
I20260709 13:00:43.076093  7615 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429"
dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
 from {username='slave'} at 127.0.0.1:59722
I20260709 13:00:43.076617  7615 raft_consensus.cc:493] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:43.076944  7615 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:43.082053  7615 raft_consensus.cc:515] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.083616  7615 leader_election.cc:290] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 4 election: Requested vote from peers b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459), 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67:37871)
I20260709 13:00:43.084641  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:43.084916  7765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e91e12d6280746fd8de26cccf5330429" candidate_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
I20260709 13:00:43.085417  7691 raft_consensus.cc:3055] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:43.085584  7765 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:43.085851  7691 raft_consensus.cc:740] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:43.086892  7691 consensus_queue.cc:260] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.088279  7691 raft_consensus.cc:3060] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:43.093771  7765 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 4.
I20260709 13:00:43.095346  7580 leader_election.cc:304] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:43.096201  7691 raft_consensus.cc:2468] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a791c55d7d0a4d91a10e02f61f7a1662 in term 4.
I20260709 13:00:43.096276  7946 raft_consensus.cc:2804] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:43.096994  7946 raft_consensus.cc:697] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 LEADER]: Becoming Leader. State: Replica: a791c55d7d0a4d91a10e02f61f7a1662, State: Running, Role: LEADER
I20260709 13:00:43.097822  7946 consensus_queue.cc:237] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.104312  7512 catalog_manager.cc:5697] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 reported cstate change: term changed from 3 to 4, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65). New cstate: current_term: 4 leader_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:43.127956  7945 raft_consensus.cc:993] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20: : Instructing follower 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:43.128424  7877 raft_consensus.cc:1081] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Signalling peer 76ae0ee745734d5e9088ff6563d716d8 to start an election
I20260709 13:00:43.129920  7765 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6"
dest_uuid: "76ae0ee745734d5e9088ff6563d716d8"
 from {username='slave'} at 127.0.0.1:53942
I20260709 13:00:43.130429  7765 raft_consensus.cc:493] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:43.130722  7765 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:43.135192  7765 raft_consensus.cc:515] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.136819  7765 leader_election.cc:290] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 4 election: Requested vote from peers a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171), b523fc673cf34021a277e15cd85c3b20 (127.7.78.66:38459)
I20260709 13:00:43.137840  7615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a791c55d7d0a4d91a10e02f61f7a1662"
I20260709 13:00:43.137843  7691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3b6ec9f23a94abea53c55ef57a0fac6" candidate_uuid: "76ae0ee745734d5e9088ff6563d716d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b523fc673cf34021a277e15cd85c3b20"
I20260709 13:00:43.138593  7615 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:43.138593  7691 raft_consensus.cc:3055] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:43.139071  7691 raft_consensus.cc:740] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b523fc673cf34021a277e15cd85c3b20, State: Running, Role: LEADER
I20260709 13:00:43.139901  7691 consensus_queue.cc:260] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.141095  7691 raft_consensus.cc:3060] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:43.145190  7615 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 4.
I20260709 13:00:43.146514  7689 raft_consensus.cc:1275] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:43.146526  7729 leader_election.cc:304] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76ae0ee745734d5e9088ff6563d716d8, a791c55d7d0a4d91a10e02f61f7a1662; no voters: 
I20260709 13:00:43.147574  7926 raft_consensus.cc:2804] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:43.147862  7691 raft_consensus.cc:2468] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ae0ee745734d5e9088ff6563d716d8 in term 4.
I20260709 13:00:43.148078  7926 raft_consensus.cc:697] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 LEADER]: Becoming Leader. State: Replica: 76ae0ee745734d5e9088ff6563d716d8, State: Running, Role: LEADER
I20260709 13:00:43.148030  7946 consensus_queue.cc:1048] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:43.149142  7926 consensus_queue.cc:237] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } }
I20260709 13:00:43.160900  7765 raft_consensus.cc:1275] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:43.157897  7512 catalog_manager.cc:5697] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 reported cstate change: term changed from 3 to 4, leader changed from b523fc673cf34021a277e15cd85c3b20 (127.7.78.66) to 76ae0ee745734d5e9088ff6563d716d8 (127.7.78.67). New cstate: current_term: 4 leader_uuid: "76ae0ee745734d5e9088ff6563d716d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:43.162489  7960 consensus_queue.cc:1048] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:43.507340  7691 raft_consensus.cc:1275] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:43.508540  7960 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:43.513954  7689 raft_consensus.cc:1275] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:43.515231  7960 consensus_queue.cc:1048] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:43.520365  7765 raft_consensus.cc:1275] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:43.523095  7946 consensus_queue.cc:1048] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:43.529167  7689 raft_consensus.cc:1275] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:43.530686  7926 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:43.533540  7764 raft_consensus.cc:1275] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Refusing update from remote peer a791c55d7d0a4d91a10e02f61f7a1662: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:43.536362  7947 consensus_queue.cc:1048] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ae0ee745734d5e9088ff6563d716d8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:43.547924  7615 raft_consensus.cc:1275] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:43.551718  7953 consensus_queue.cc:1048] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:43.676384  7691 raft_consensus.cc:1275] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:43.677752  7926 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523fc673cf34021a277e15cd85c3b20" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:43.678228  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:43.688155  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
I20260709 13:00:43.691062  7615 raft_consensus.cc:1275] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Refusing update from remote peer 76ae0ee745734d5e9088ff6563d716d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:43.693207  7953 consensus_queue.cc:1048] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a791c55d7d0a4d91a10e02f61f7a1662" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:00:43.697052  7729 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:00:43.702696  7729 consensus_peers.cc:597] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 -> Peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Couldn't send request to peer a791c55d7d0a4d91a10e02f61f7a1662. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:43.725078  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:43.726043  7481 tablet_replica.cc:333] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.726678  7481 raft_consensus.cc:2243] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.727420  7481 raft_consensus.cc:2272] T 2f4cb96a3fc044fd92d7be13e0807e60 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.730520  7481 tablet_replica.cc:333] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.731010  7481 raft_consensus.cc:2243] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.731406  7481 raft_consensus.cc:2272] T 28f891b1555d4bdf94235c996aef9354 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.732966  7481 tablet_replica.cc:333] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.733577  7481 raft_consensus.cc:2243] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.733935  7481 raft_consensus.cc:2272] T 968f7bdc989246688fc4e10e8fd870a2 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.736538  7481 tablet_replica.cc:333] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.736995  7481 raft_consensus.cc:2243] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.737365  7481 raft_consensus.cc:2272] T f3128520a7c344758ca9dac16999e350 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.739017  7481 tablet_replica.cc:333] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.739439  7481 raft_consensus.cc:2243] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.739796  7481 raft_consensus.cc:2272] T 1d44fab36da747c08f6c6b34bba33782 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.741266  7481 tablet_replica.cc:333] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.741771  7481 raft_consensus.cc:2243] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.742144  7481 raft_consensus.cc:2272] T 4e834a2c2ef24568a8505066a5b8bdd3 P a791c55d7d0a4d91a10e02f61f7a1662 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.743584  7481 tablet_replica.cc:333] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.743993  7481 raft_consensus.cc:2243] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.744663  7481 raft_consensus.cc:2272] T e91e12d6280746fd8de26cccf5330429 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.747088  7481 tablet_replica.cc:333] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.747575  7481 raft_consensus.cc:2243] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.748338  7481 raft_consensus.cc:2272] T e5d74af98f944e3ca1510f01c68ab683 P a791c55d7d0a4d91a10e02f61f7a1662 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.750411  7481 tablet_replica.cc:333] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662: stopping tablet replica
I20260709 13:00:43.750886  7481 raft_consensus.cc:2243] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.751395  7481 raft_consensus.cc:2272] T c3b6ec9f23a94abea53c55ef57a0fac6 P a791c55d7d0a4d91a10e02f61f7a1662 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:00:43.754676  7655 consensus_peers.cc:597] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 -> Peer a791c55d7d0a4d91a10e02f61f7a1662 (127.7.78.65:43171): Couldn't send request to peer a791c55d7d0a4d91a10e02f61f7a1662. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:43.851135  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
I20260709 13:00:43.870246  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
I20260709 13:00:43.896095  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:43.896791  7481 tablet_replica.cc:333] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.897778  7481 raft_consensus.cc:2243] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.898377  7481 raft_consensus.cc:2272] T 2f4cb96a3fc044fd92d7be13e0807e60 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.900563  7481 tablet_replica.cc:333] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.901103  7481 raft_consensus.cc:2243] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.901592  7481 raft_consensus.cc:2272] T 968f7bdc989246688fc4e10e8fd870a2 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.903852  7481 tablet_replica.cc:333] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.904362  7481 raft_consensus.cc:2243] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.905067  7481 raft_consensus.cc:2272] T f3128520a7c344758ca9dac16999e350 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.906800  7481 tablet_replica.cc:333] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.907248  7481 raft_consensus.cc:2243] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.907598  7481 raft_consensus.cc:2272] T c3b6ec9f23a94abea53c55ef57a0fac6 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.909744  7481 tablet_replica.cc:333] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.910306  7481 raft_consensus.cc:2243] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.910714  7481 raft_consensus.cc:2272] T e5d74af98f944e3ca1510f01c68ab683 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.913056  7481 tablet_replica.cc:333] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.913640  7481 raft_consensus.cc:2243] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.914047  7481 raft_consensus.cc:2272] T e91e12d6280746fd8de26cccf5330429 P b523fc673cf34021a277e15cd85c3b20 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.916149  7481 tablet_replica.cc:333] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.916612  7481 raft_consensus.cc:2243] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.917007  7481 raft_consensus.cc:2272] T 28f891b1555d4bdf94235c996aef9354 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.918641  7481 tablet_replica.cc:333] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.919059  7481 raft_consensus.cc:2243] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.919709  7481 raft_consensus.cc:2272] T 4e834a2c2ef24568a8505066a5b8bdd3 P b523fc673cf34021a277e15cd85c3b20 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.921303  7481 tablet_replica.cc:333] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20: stopping tablet replica
I20260709 13:00:43.921756  7481 raft_consensus.cc:2243] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.922446  7481 raft_consensus.cc:2272] T 1d44fab36da747c08f6c6b34bba33782 P b523fc673cf34021a277e15cd85c3b20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.942538  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
I20260709 13:00:43.959324  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
I20260709 13:00:43.987068  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:43.987884  7481 tablet_replica.cc:333] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:43.988771  7481 raft_consensus.cc:2243] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.989756  7481 raft_consensus.cc:2272] T 28f891b1555d4bdf94235c996aef9354 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.991423  7481 tablet_replica.cc:333] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:43.991912  7481 raft_consensus.cc:2243] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.992314  7481 raft_consensus.cc:2272] T e91e12d6280746fd8de26cccf5330429 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.994807  7481 tablet_replica.cc:333] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:43.995322  7481 raft_consensus.cc:2243] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:43.995698  7481 raft_consensus.cc:2272] T e5d74af98f944e3ca1510f01c68ab683 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:43.997802  7481 tablet_replica.cc:333] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:43.998255  7481 raft_consensus.cc:2243] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:43.998944  7481 raft_consensus.cc:2272] T c3b6ec9f23a94abea53c55ef57a0fac6 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:44.000736  7481 tablet_replica.cc:333] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:44.001183  7481 raft_consensus.cc:2243] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:44.001691  7481 raft_consensus.cc:2272] T f3128520a7c344758ca9dac16999e350 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:44.003288  7481 tablet_replica.cc:333] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:44.003697  7481 raft_consensus.cc:2243] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:44.004379  7481 raft_consensus.cc:2272] T 968f7bdc989246688fc4e10e8fd870a2 P 76ae0ee745734d5e9088ff6563d716d8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:44.006330  7481 tablet_replica.cc:333] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:44.006805  7481 raft_consensus.cc:2243] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:44.007200  7481 raft_consensus.cc:2272] T 1d44fab36da747c08f6c6b34bba33782 P 76ae0ee745734d5e9088ff6563d716d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:44.008703  7481 tablet_replica.cc:333] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:44.009109  7481 raft_consensus.cc:2243] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:44.009569  7481 raft_consensus.cc:2272] T 4e834a2c2ef24568a8505066a5b8bdd3 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:44.011212  7481 tablet_replica.cc:333] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8: stopping tablet replica
I20260709 13:00:44.011602  7481 raft_consensus.cc:2243] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:44.011932  7481 raft_consensus.cc:2272] T 2f4cb96a3fc044fd92d7be13e0807e60 P 76ae0ee745734d5e9088ff6563d716d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:44.038434  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
I20260709 13:00:44.054930  7481 master.cc:562] Master@127.7.78.126:39787 shutting down...
I20260709 13:00:44.261006  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:44.261624  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:44.261942  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P 949ee267387b4cd2b7b81de073bf36b8: stopping tablet replica
I20260709 13:00:44.281885  7481 master.cc:584] Master@127.7.78.126:39787 shutdown complete.
76ae0ee745734d5e9088ff6563d716d8  1
a791c55d7d0a4d91a10e02f61f7a1662  4
b523fc673cf34021a277e15cd85c3b20  4
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (12746 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 13:00:44.323382  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:39041
I20260709 13:00:44.324795  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:44.331250  7974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:44.331547  7975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:44.332921  7977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:44.332953  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:44.334054  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:44.334268  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:44.334431  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602044334414 us; error 0 us; skew 500 ppm
I20260709 13:00:44.334955  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:44.337566  7481 webserver.cc:533] Webserver started at http://127.7.78.126:36961/ using document root <none> and password file <none>
I20260709 13:00:44.338063  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:44.338270  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:44.338582  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:44.339725  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "879881d88c0d4f29b35d2c1cad7a8fd9"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.344425  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20260709 13:00:44.348088  7982 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.349097  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:00:44.349491  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "879881d88c0d4f29b35d2c1cad7a8fd9"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.349802  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:44.385809  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:44.387001  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:44.425385  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:39041
I20260709 13:00:44.425535  8033 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:39041 every 8 connection(s)
I20260709 13:00:44.429555  8034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:44.439118  8034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9: Bootstrap starting.
I20260709 13:00:44.443894  8034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.448282  8034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9: No bootstrap required, opened a new log
I20260709 13:00:44.450664  8034 raft_consensus.cc:359] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" member_type: VOTER }
I20260709 13:00:44.451076  8034 raft_consensus.cc:385] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.451360  8034 raft_consensus.cc:740] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879881d88c0d4f29b35d2c1cad7a8fd9, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.451982  8034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" member_type: VOTER }
I20260709 13:00:44.452391  8034 raft_consensus.cc:399] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:00:44.452646  8034 raft_consensus.cc:493] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:00:44.452944  8034 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.457093  8034 raft_consensus.cc:515] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" member_type: VOTER }
I20260709 13:00:44.457805  8034 leader_election.cc:304] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 879881d88c0d4f29b35d2c1cad7a8fd9; no voters: 
I20260709 13:00:44.459015  8034 leader_election.cc:290] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:00:44.459543  8037 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:44.461018  8037 raft_consensus.cc:697] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 1 LEADER]: Becoming Leader. State: Replica: 879881d88c0d4f29b35d2c1cad7a8fd9, State: Running, Role: LEADER
I20260709 13:00:44.461983  8034 sys_catalog.cc:565] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:00:44.461855  8037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" member_type: VOTER }
I20260709 13:00:44.469197  8039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 879881d88c0d4f29b35d2c1cad7a8fd9. Latest consensus state: current_term: 1 leader_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" member_type: VOTER } }
I20260709 13:00:44.469913  8039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:44.469892  8038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879881d88c0d4f29b35d2c1cad7a8fd9" member_type: VOTER } }
I20260709 13:00:44.470597  8038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:44.472077  8045 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:00:44.477500  8045 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:00:44.480098  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:00:44.487692  8045 catalog_manager.cc:1383] Generated new cluster ID: 953af3b89c784cf3901850c5377cdbe7
I20260709 13:00:44.488039  8045 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:00:44.510551  8045 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:00:44.512171  8045 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:00:44.534765  8045 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9: Generated new TSK 0
I20260709 13:00:44.535531  8045 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:00:44.551628  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:44.557963  8055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:44.558848  8056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:44.562052  8058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:44.562120  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:44.563184  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:44.563408  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:44.563659  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602044563639 us; error 0 us; skew 500 ppm
I20260709 13:00:44.564210  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:44.566644  7481 webserver.cc:533] Webserver started at http://127.7.78.65:43459/ using document root <none> and password file <none>
I20260709 13:00:44.567204  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:44.567391  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:44.567682  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:44.568722  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.573541  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:00:44.577029  8063 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.578018  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:00:44.578337  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.578680  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:44.607825  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:44.609166  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:44.610755  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:44.613193  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:44.613440  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.613703  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:44.613901  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.659603  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:33213
I20260709 13:00:44.659700  8125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:33213 every 8 connection(s)
I20260709 13:00:44.664425  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:44.672619  8130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:44.673293  8131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:44.677873  8133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:44.678051  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:44.679626  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:44.679977  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:44.680207  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602044680187 us; error 0 us; skew 500 ppm
I20260709 13:00:44.681046  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:44.683307  8126 heartbeater.cc:344] Connected to a master server at 127.7.78.126:39041
I20260709 13:00:44.683705  8126 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:44.684623  8126 heartbeater.cc:507] Master 127.7.78.126:39041 requested a full tablet report, sending...
I20260709 13:00:44.684724  7481 webserver.cc:533] Webserver started at http://127.7.78.66:41569/ using document root <none> and password file <none>
I20260709 13:00:44.685349  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:44.685643  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:44.686002  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:44.687018  7999 ts_manager.cc:194] Registered new tserver with Master: fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213)
I20260709 13:00:44.687412  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "f9eb3440010246a59b2fe21ddc2736cd"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.689081  7999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46030
I20260709 13:00:44.692648  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20260709 13:00:44.696671  8138 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.697700  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:00:44.698063  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "f9eb3440010246a59b2fe21ddc2736cd"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.698541  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:44.725358  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:44.726630  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:44.728174  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:44.730453  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:44.730708  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.730955  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:44.731127  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.772548  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:34305
I20260709 13:00:44.772657  8200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:34305 every 8 connection(s)
I20260709 13:00:44.777240  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:44.786877  8204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:44.788048  8205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:44.792737  8201 heartbeater.cc:344] Connected to a master server at 127.7.78.126:39041
I20260709 13:00:44.793329  8201 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:44.794061  7481 server_base.cc:1061] running on GCE node
W20260709 13:00:44.794150  8207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:44.794378  8201 heartbeater.cc:507] Master 127.7.78.126:39041 requested a full tablet report, sending...
I20260709 13:00:44.795354  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:44.795632  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:44.795841  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602044795818 us; error 0 us; skew 500 ppm
I20260709 13:00:44.796509  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:44.796978  7999 ts_manager.cc:194] Registered new tserver with Master: f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:44.798619  7999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46042
I20260709 13:00:44.801515  7481 webserver.cc:533] Webserver started at http://127.7.78.67:44613/ using document root <none> and password file <none>
I20260709 13:00:44.802027  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:44.802222  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:44.802502  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:44.803622  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.808116  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:00:44.811569  8212 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.812484  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 13:00:44.812747  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
format_stamp: "Formatted at 2026-07-09 13:00:44 on dist-test-slave-50cs"
I20260709 13:00:44.812989  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:44.844703  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:44.846122  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:44.847817  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:44.850097  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:44.850315  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.850522  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:44.850663  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:44.898171  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:37383
I20260709 13:00:44.898464  8274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:37383 every 8 connection(s)
I20260709 13:00:44.912933  8275 heartbeater.cc:344] Connected to a master server at 127.7.78.126:39041
I20260709 13:00:44.913367  8275 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:44.914346  8275 heartbeater.cc:507] Master 127.7.78.126:39041 requested a full tablet report, sending...
I20260709 13:00:44.916523  7999 ts_manager.cc:194] Registered new tserver with Master: c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:44.917534  7481 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016021723s
I20260709 13:00:44.918140  7999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46044
I20260709 13:00:44.944761  7999 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46058:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:00:44.947810  7999 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:00:45.036170  8166 tablet_service.cc:1511] Processing CreateTablet for tablet d7671f95809843818836fbe6cd9076ef (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:00:45.038599  8165 tablet_service.cc:1511] Processing CreateTablet for tablet 5ff8b45bea7f4af481db696cab24dac3 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:00:45.041241  8164 tablet_service.cc:1511] Processing CreateTablet for tablet a5d641112e2a4f6ea3dea8fbbc499f64 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:00:45.042691  8163 tablet_service.cc:1511] Processing CreateTablet for tablet 7d843ab104dc4b39a7ef8c0f01dc62a4 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:00:45.045775  8166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7671f95809843818836fbe6cd9076ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.046192  8162 tablet_service.cc:1511] Processing CreateTablet for tablet f0d428e367ce4dffa8d66f0deb321f6a (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:00:45.046886  8165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ff8b45bea7f4af481db696cab24dac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.047689  8162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d428e367ce4dffa8d66f0deb321f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.048349  8164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d641112e2a4f6ea3dea8fbbc499f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.048276  8161 tablet_service.cc:1511] Processing CreateTablet for tablet 6210b6863d4e474f9dec78996ab0f808 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:00:45.049896  8161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6210b6863d4e474f9dec78996ab0f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.050123  8160 tablet_service.cc:1511] Processing CreateTablet for tablet 0a4985df633044ffa4bb78d154489b51 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:00:45.051600  8160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a4985df633044ffa4bb78d154489b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.054776  8159 tablet_service.cc:1511] Processing CreateTablet for tablet bcd7704764cb487ba1975b42e16e384f (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:00:45.056407  8159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd7704764cb487ba1975b42e16e384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.057789  8163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d843ab104dc4b39a7ef8c0f01dc62a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.060526  8158 tablet_service.cc:1511] Processing CreateTablet for tablet 7dff4362b5f14c30ab3505adf6e5e89f (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:00:45.062130  8158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dff4362b5f14c30ab3505adf6e5e89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.068377  8157 tablet_service.cc:1511] Processing CreateTablet for tablet d57723b298b34d6f889d10aa14cd4094 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:00:45.070158  8157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57723b298b34d6f889d10aa14cd4094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.074144  8240 tablet_service.cc:1511] Processing CreateTablet for tablet d7671f95809843818836fbe6cd9076ef (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:00:45.075591  8240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7671f95809843818836fbe6cd9076ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.075457  8236 tablet_service.cc:1511] Processing CreateTablet for tablet f0d428e367ce4dffa8d66f0deb321f6a (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:00:45.077170  8236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d428e367ce4dffa8d66f0deb321f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.078680  8239 tablet_service.cc:1511] Processing CreateTablet for tablet 5ff8b45bea7f4af481db696cab24dac3 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:00:45.080049  8239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ff8b45bea7f4af481db696cab24dac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.081681  8238 tablet_service.cc:1511] Processing CreateTablet for tablet a5d641112e2a4f6ea3dea8fbbc499f64 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:00:45.083571  8238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d641112e2a4f6ea3dea8fbbc499f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.085677  8235 tablet_service.cc:1511] Processing CreateTablet for tablet 6210b6863d4e474f9dec78996ab0f808 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:00:45.087621  8235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6210b6863d4e474f9dec78996ab0f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.074086  8237 tablet_service.cc:1511] Processing CreateTablet for tablet 7d843ab104dc4b39a7ef8c0f01dc62a4 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:00:45.091127  8237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d843ab104dc4b39a7ef8c0f01dc62a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.094038  8234 tablet_service.cc:1511] Processing CreateTablet for tablet 0a4985df633044ffa4bb78d154489b51 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:00:45.097620  8234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a4985df633044ffa4bb78d154489b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.102053  8233 tablet_service.cc:1511] Processing CreateTablet for tablet bcd7704764cb487ba1975b42e16e384f (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:00:45.103515  8233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd7704764cb487ba1975b42e16e384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.107753  8232 tablet_service.cc:1511] Processing CreateTablet for tablet 7dff4362b5f14c30ab3505adf6e5e89f (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:00:45.110329  8232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dff4362b5f14c30ab3505adf6e5e89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.111321  8231 tablet_service.cc:1511] Processing CreateTablet for tablet d57723b298b34d6f889d10aa14cd4094 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:00:45.112839  8231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57723b298b34d6f889d10aa14cd4094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.154985  8091 tablet_service.cc:1511] Processing CreateTablet for tablet d7671f95809843818836fbe6cd9076ef (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:00:45.155108  8090 tablet_service.cc:1511] Processing CreateTablet for tablet 5ff8b45bea7f4af481db696cab24dac3 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:00:45.156976  8091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7671f95809843818836fbe6cd9076ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.157891  8089 tablet_service.cc:1511] Processing CreateTablet for tablet a5d641112e2a4f6ea3dea8fbbc499f64 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:00:45.159549  8089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d641112e2a4f6ea3dea8fbbc499f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.160429  8090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ff8b45bea7f4af481db696cab24dac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.167289  8088 tablet_service.cc:1511] Processing CreateTablet for tablet 7d843ab104dc4b39a7ef8c0f01dc62a4 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:00:45.171779  8087 tablet_service.cc:1511] Processing CreateTablet for tablet f0d428e367ce4dffa8d66f0deb321f6a (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:00:45.169274  8088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d843ab104dc4b39a7ef8c0f01dc62a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.177886  8087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d428e367ce4dffa8d66f0deb321f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.187460  8086 tablet_service.cc:1511] Processing CreateTablet for tablet 6210b6863d4e474f9dec78996ab0f808 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:00:45.189728  8086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6210b6863d4e474f9dec78996ab0f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.213212  8084 tablet_service.cc:1511] Processing CreateTablet for tablet bcd7704764cb487ba1975b42e16e384f (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:00:45.214715  8084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd7704764cb487ba1975b42e16e384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.222545  8083 tablet_service.cc:1511] Processing CreateTablet for tablet 7dff4362b5f14c30ab3505adf6e5e89f (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:00:45.224525  8083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dff4362b5f14c30ab3505adf6e5e89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.212955  8082 tablet_service.cc:1511] Processing CreateTablet for tablet d57723b298b34d6f889d10aa14cd4094 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:00:45.227401  8082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57723b298b34d6f889d10aa14cd4094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.212931  8085 tablet_service.cc:1511] Processing CreateTablet for tablet 0a4985df633044ffa4bb78d154489b51 (DEFAULT_TABLE table=test-workload [id=90cc98985c3845c99a7f090033a03f70]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:00:45.233749  8085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a4985df633044ffa4bb78d154489b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:45.251649  8291 tablet_bootstrap.cc:492] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:45.278869  8291 tablet_bootstrap.cc:654] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.286144  8290 tablet_bootstrap.cc:492] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.319581  8293 tablet_bootstrap.cc:492] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.331239  8290 tablet_bootstrap.cc:654] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.336594  8291 tablet_bootstrap.cc:492] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:45.337270  8291 ts_tablet_manager.cc:1403] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.086s	user 0.012s	sys 0.010s
I20260709 13:00:45.340582  8293 tablet_bootstrap.cc:654] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.340230  8291 raft_consensus.cc:359] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.340996  8291 raft_consensus.cc:385] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.341318  8291 raft_consensus.cc:740] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.342288  8291 consensus_queue.cc:260] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.345309  8290 tablet_bootstrap.cc:492] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.345835  8290 ts_tablet_manager.cc:1403] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.060s	user 0.009s	sys 0.003s
I20260709 13:00:45.348055  8290 raft_consensus.cc:359] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.348584  8290 raft_consensus.cc:385] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.348917  8290 raft_consensus.cc:740] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.349453  8275 heartbeater.cc:499] Master 127.7.78.126:39041 was elected leader, sending a full tablet report...
I20260709 13:00:45.349846  8290 consensus_queue.cc:260] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.351364  8291 ts_tablet_manager.cc:1434] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.004s
I20260709 13:00:45.353123  8291 tablet_bootstrap.cc:492] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:45.357182  8201 heartbeater.cc:499] Master 127.7.78.126:39041 was elected leader, sending a full tablet report...
I20260709 13:00:45.357820  8290 ts_tablet_manager.cc:1434] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.005s
I20260709 13:00:45.358479  8293 tablet_bootstrap.cc:492] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.358815  8290 tablet_bootstrap.cc:492] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.358942  8293 ts_tablet_manager.cc:1403] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.040s	user 0.011s	sys 0.012s
I20260709 13:00:45.362051  8293 raft_consensus.cc:359] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.362826  8293 raft_consensus.cc:385] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.363219  8293 raft_consensus.cc:740] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.364115  8293 consensus_queue.cc:260] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.365775  8290 tablet_bootstrap.cc:654] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.371570  8291 tablet_bootstrap.cc:654] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.394868  8126 heartbeater.cc:499] Master 127.7.78.126:39041 was elected leader, sending a full tablet report...
I20260709 13:00:45.395412  8293 ts_tablet_manager.cc:1434] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.036s	user 0.025s	sys 0.011s
I20260709 13:00:45.396708  8293 tablet_bootstrap.cc:492] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.404341  8293 tablet_bootstrap.cc:654] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.412096  8290 tablet_bootstrap.cc:492] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.412710  8290 ts_tablet_manager.cc:1403] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.054s	user 0.013s	sys 0.024s
I20260709 13:00:45.416872  8291 tablet_bootstrap.cc:492] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:45.416738  8290 raft_consensus.cc:359] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.417461  8291 ts_tablet_manager.cc:1403] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.065s	user 0.023s	sys 0.022s
I20260709 13:00:45.417527  8290 raft_consensus.cc:385] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.417960  8290 raft_consensus.cc:740] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.418866  8290 consensus_queue.cc:260] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.421077  8290 ts_tablet_manager.cc:1434] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 13:00:45.420718  8291 raft_consensus.cc:359] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.421523  8291 raft_consensus.cc:385] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.421845  8291 raft_consensus.cc:740] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.422078  8290 tablet_bootstrap.cc:492] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.422808  8291 consensus_queue.cc:260] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.418864  8293 tablet_bootstrap.cc:492] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.425552  8293 ts_tablet_manager.cc:1403] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.016s
I20260709 13:00:45.428040  8291 ts_tablet_manager.cc:1434] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.002s
I20260709 13:00:45.428707  8293 raft_consensus.cc:359] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.429298  8291 tablet_bootstrap.cc:492] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:45.429550  8293 raft_consensus.cc:385] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.430073  8290 tablet_bootstrap.cc:654] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.430006  8293 raft_consensus.cc:740] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.430909  8293 consensus_queue.cc:260] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.433887  8293 ts_tablet_manager.cc:1434] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 13:00:45.435281  8293 tablet_bootstrap.cc:492] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.438351  8291 tablet_bootstrap.cc:654] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.443974  8290 tablet_bootstrap.cc:492] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.444514  8290 ts_tablet_manager.cc:1403] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.008s
I20260709 13:00:45.447839  8290 raft_consensus.cc:359] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.447842  8293 tablet_bootstrap.cc:654] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.448717  8290 raft_consensus.cc:385] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.449255  8290 raft_consensus.cc:740] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.450146  8290 consensus_queue.cc:260] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.450946  8295 raft_consensus.cc:493] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.451542  8295 raft_consensus.cc:515] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.452860  8290 ts_tablet_manager.cc:1434] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.005s
I20260709 13:00:45.453979  8290 tablet_bootstrap.cc:492] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.454978  8291 tablet_bootstrap.cc:492] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:45.456014  8291 ts_tablet_manager.cc:1403] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.006s
I20260709 13:00:45.459489  8291 raft_consensus.cc:359] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.460729  8295 leader_election.cc:290] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:45.461431  8290 tablet_bootstrap.cc:654] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.461722  8291 raft_consensus.cc:385] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.462097  8291 raft_consensus.cc:740] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.463016  8291 consensus_queue.cc:260] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.465834  8291 ts_tablet_manager.cc:1434] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.006s
I20260709 13:00:45.467376  8291 tablet_bootstrap.cc:492] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
W20260709 13:00:45.475459  8047 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:45.485040  8291 tablet_bootstrap.cc:654] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.487118  8290 tablet_bootstrap.cc:492] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.487797  8290 ts_tablet_manager.cc:1403] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.009s
I20260709 13:00:45.491765  8290 raft_consensus.cc:359] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.492789  8290 raft_consensus.cc:385] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.493158  8290 raft_consensus.cc:740] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.494167  8290 consensus_queue.cc:260] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.495783  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:45.497956  8295 raft_consensus.cc:493] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.498538  8295 raft_consensus.cc:515] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.500487  8290 ts_tablet_manager.cc:1434] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.004s
I20260709 13:00:45.500985  8295 leader_election.cc:290] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:45.501953  8290 tablet_bootstrap.cc:492] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
W20260709 13:00:45.502350  8215 leader_election.cc:343] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.506055  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
W20260709 13:00:45.507931  8215 leader_election.cc:343] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.510825  8290 tablet_bootstrap.cc:654] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.512141  8293 tablet_bootstrap.cc:492] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.513221  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.513962  8176 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 0.
I20260709 13:00:45.514289  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.512631  8293 ts_tablet_manager.cc:1403] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.078s	user 0.014s	sys 0.023s
I20260709 13:00:45.515810  8291 tablet_bootstrap.cc:492] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:45.515594  8213 leader_election.cc:304] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.516290  8291 ts_tablet_manager.cc:1403] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.049s	user 0.020s	sys 0.001s
I20260709 13:00:45.516884  8295 raft_consensus.cc:2804] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.517324  8295 raft_consensus.cc:493] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.517756  8295 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.518600  8293 raft_consensus.cc:359] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.519547  8293 raft_consensus.cc:385] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.519487  8291 raft_consensus.cc:359] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.520569  8291 raft_consensus.cc:385] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.520956  8291 raft_consensus.cc:740] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
W20260709 13:00:45.521143  8213 leader_election.cc:343] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.521814  8213 leader_election.cc:304] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2; no voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.520078  8293 raft_consensus.cc:740] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.522042  8291 consensus_queue.cc:260] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.523861  8293 consensus_queue.cc:260] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.525857  8295 raft_consensus.cc:515] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.528681  8295 leader_election.cc:290] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:45.529748  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:45.532899  8295 raft_consensus.cc:2749] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:45.534345  8215 leader_election.cc:343] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.526410  8293 ts_tablet_manager.cc:1434] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 13:00:45.537627  8297 raft_consensus.cc:493] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.538170  8297 raft_consensus.cc:515] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.538702  8293 tablet_bootstrap.cc:492] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.546928  8297 leader_election.cc:290] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.547233  8293 tablet_bootstrap.cc:654] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.547420  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:45.548127  8175 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.559477  8175 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 1.
I20260709 13:00:45.561854  8290 tablet_bootstrap.cc:492] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.562673  8290 ts_tablet_manager.cc:1403] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.061s	user 0.019s	sys 0.002s
I20260709 13:00:45.566246  8290 raft_consensus.cc:359] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.561635  8213 leader_election.cc:304] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.567485  8310 raft_consensus.cc:2804] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.568028  8310 raft_consensus.cc:697] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Becoming Leader. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:45.568912  8310 consensus_queue.cc:237] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.570576  8290 raft_consensus.cc:385] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.570883  8293 tablet_bootstrap.cc:492] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.571029  8290 raft_consensus.cc:740] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.571573  8293 ts_tablet_manager.cc:1403] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.033s	user 0.007s	sys 0.023s
I20260709 13:00:45.572203  8290 consensus_queue.cc:260] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.574968  8290 ts_tablet_manager.cc:1434] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.003s
I20260709 13:00:45.574847  8293 raft_consensus.cc:359] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.575742  8293 raft_consensus.cc:385] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.576107  8293 raft_consensus.cc:740] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.576313  8290 tablet_bootstrap.cc:492] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.577138  8293 consensus_queue.cc:260] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.584645  8290 tablet_bootstrap.cc:654] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.585700  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:45.586496  8101 raft_consensus.cc:2468] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:45.587901  8293 ts_tablet_manager.cc:1434] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.003s
I20260709 13:00:45.588058  8141 leader_election.cc:304] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:45.589136  8297 raft_consensus.cc:2804] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.589583  8297 raft_consensus.cc:493] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.589974  8297 raft_consensus.cc:3060] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.590667  8291 ts_tablet_manager.cc:1434] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.074s	user 0.017s	sys 0.012s
I20260709 13:00:45.594326  8291 tablet_bootstrap.cc:492] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:45.598034  8297 raft_consensus.cc:515] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.598683  8293 tablet_bootstrap.cc:492] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.607223  8250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:45.609465  8293 tablet_bootstrap.cc:654] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:45.609582  8142 leader_election.cc:343] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.612972  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:45.613770  8101 raft_consensus.cc:3060] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.606339  8291 tablet_bootstrap.cc:654] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.621058  8298 raft_consensus.cc:493] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.622078  8298 raft_consensus.cc:515] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.621167  7996 catalog_manager.cc:5697] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:45.629653  8291 tablet_bootstrap.cc:492] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:45.630339  8291 ts_tablet_manager.cc:1403] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.036s	user 0.008s	sys 0.011s
I20260709 13:00:45.631680  8101 raft_consensus.cc:2468] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:45.633941  8291 raft_consensus.cc:359] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.634833  8291 raft_consensus.cc:385] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.634739  8250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:45.635301  8297 leader_election.cc:290] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.636348  8298 leader_election.cc:290] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.638787  8290 tablet_bootstrap.cc:492] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.642624  8291 raft_consensus.cc:740] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.642952  8290 ts_tablet_manager.cc:1403] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.067s	user 0.033s	sys 0.027s
I20260709 13:00:45.639128  8141 leader_election.cc:304] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:45.644120  8297 raft_consensus.cc:2804] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.643534  8291 consensus_queue.cc:260] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
W20260709 13:00:45.644903  8142 leader_election.cc:343] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.650693  8293 tablet_bootstrap.cc:492] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.651415  8293 ts_tablet_manager.cc:1403] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.053s	user 0.019s	sys 0.005s
I20260709 13:00:45.655718  8290 raft_consensus.cc:359] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.656570  8290 raft_consensus.cc:385] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.656980  8291 ts_tablet_manager.cc:1434] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 13:00:45.656920  8290 raft_consensus.cc:740] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.657588  8298 raft_consensus.cc:493] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.657876  8290 consensus_queue.cc:260] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.654881  8297 raft_consensus.cc:697] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:45.658277  8298 raft_consensus.cc:515] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.658174  8291 tablet_bootstrap.cc:492] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:45.658733  8293 raft_consensus.cc:359] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.663127  8293 raft_consensus.cc:385] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.663683  8293 raft_consensus.cc:740] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.664511  8293 consensus_queue.cc:260] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.661676  8297 consensus_queue.cc:237] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.666508  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.667302  8324 raft_consensus.cc:493] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.668303  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.669094  8298 leader_election.cc:290] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
W20260709 13:00:45.671401  8064 leader_election.cc:343] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.672279  8324 raft_consensus.cc:515] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.673730  8176 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 0.
I20260709 13:00:45.678442  8064 leader_election.cc:304] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:45.679900  8298 raft_consensus.cc:2804] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.680428  8298 raft_consensus.cc:493] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.680863  8298 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.685293  8293 ts_tablet_manager.cc:1434] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.033s	user 0.011s	sys 0.000s
I20260709 13:00:45.689097  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:45.689785  8250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:45.690647  8293 tablet_bootstrap.cc:492] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.690716  8250 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:45.692651  8250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:45.693022  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:45.693368  8250 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 0.
I20260709 13:00:45.694000  8249 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 0.
I20260709 13:00:45.694962  8067 leader_election.cc:304] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:45.695494  8142 leader_election.cc:304] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:45.696550  8328 raft_consensus.cc:2804] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.697715  8324 leader_election.cc:290] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.698431  8324 raft_consensus.cc:493] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.699719  8310 raft_consensus.cc:493] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.700305  8310 raft_consensus.cc:515] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.704548  8310 leader_election.cc:290] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213)
I20260709 13:00:45.706295  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
W20260709 13:00:45.708498  8215 leader_election.cc:343] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.709688  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.710685  8176 raft_consensus.cc:2468] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 0.
I20260709 13:00:45.697038  8328 raft_consensus.cc:493] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.712391  8213 leader_election.cc:304] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.698974  8324 raft_consensus.cc:515] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.715059  8310 raft_consensus.cc:2804] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.715633  8310 raft_consensus.cc:493] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.716152  8310 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.717569  8328 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.715175  8297 raft_consensus.cc:2804] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.721602  8297 raft_consensus.cc:493] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.722141  8297 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.723894  8329 raft_consensus.cc:493] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.724087  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:45.726476  8100 raft_consensus.cc:2468] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:45.725646  8329 raft_consensus.cc:515] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.728127  8290 ts_tablet_manager.cc:1434] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.085s	user 0.048s	sys 0.023s
I20260709 13:00:45.727941  8141 leader_election.cc:304] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:45.716137  8293 tablet_bootstrap.cc:654] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.739219  8326 raft_consensus.cc:2804] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.739557  8290 tablet_bootstrap.cc:492] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.739701  8326 raft_consensus.cc:493] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.740275  8326 raft_consensus.cc:3060] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.728730  8324 leader_election.cc:290] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.741636  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
W20260709 13:00:45.743221  8141 leader_election.cc:343] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.732594  8291 tablet_bootstrap.cc:654] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.733047  8298 raft_consensus.cc:515] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.733883  8250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:45.751845  8101 raft_consensus.cc:2393] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9eb3440010246a59b2fe21ddc2736cd in current term 1: Already voted for candidate fbbb00b12d0e4da58236f580ae8e2df9 in this term.
W20260709 13:00:45.751873  8142 leader_election.cc:343] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.754182  8290 tablet_bootstrap.cc:654] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.754220  8293 tablet_bootstrap.cc:492] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.754882  8293 ts_tablet_manager.cc:1403] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.064s	user 0.013s	sys 0.007s
I20260709 13:00:45.756531  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:45.758725  8293 raft_consensus.cc:359] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.759922  8293 raft_consensus.cc:385] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.759671  8326 raft_consensus.cc:515] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.736799  7997 catalog_manager.cc:5697] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 1 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.763235  8250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:45.763907  8250 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.764319  8329 leader_election.cc:290] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.730363  8310 raft_consensus.cc:515] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.764801  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:45.765431  8101 raft_consensus.cc:3060] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.766494  8310 leader_election.cc:290] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213)
I20260709 13:00:45.767627  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:45.767972  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:45.767246  8328 raft_consensus.cc:515] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.768540  8249 raft_consensus.cc:2468] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:45.768857  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:45.760462  8293 raft_consensus.cc:740] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.769611  8175 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.770465  8328 leader_election.cc:290] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.770717  8293 consensus_queue.cc:260] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.773355  8293 ts_tablet_manager.cc:1434] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 13:00:45.774619  8293 tablet_bootstrap.cc:492] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.775938  8174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
W20260709 13:00:45.777710  8064 leader_election.cc:343] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.779966  8142 leader_election.cc:304] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.761827  8333 raft_consensus.cc:493] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.762029  8298 leader_election.cc:290] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.735663  8297 raft_consensus.cc:515] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
W20260709 13:00:45.774514  8215 leader_election.cc:343] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.782058  8176 raft_consensus.cc:2393] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fbbb00b12d0e4da58236f580ae8e2df9 in current term 1: Already voted for candidate f9eb3440010246a59b2fe21ddc2736cd in this term.
I20260709 13:00:45.783113  8297 leader_election.cc:290] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.783761  8297 raft_consensus.cc:2804] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.784134  8297 raft_consensus.cc:493] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.784500  8297 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.785145  8293 tablet_bootstrap.cc:654] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.775940  8291 tablet_bootstrap.cc:492] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:45.785974  8291 ts_tablet_manager.cc:1403] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.128s	user 0.008s	sys 0.008s
I20260709 13:00:45.787420  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:45.793728  8333 raft_consensus.cc:515] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.795275  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:45.796219  8100 raft_consensus.cc:2393] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f9eb3440010246a59b2fe21ddc2736cd in current term 1: Already voted for candidate fbbb00b12d0e4da58236f580ae8e2df9 in this term.
I20260709 13:00:45.796531  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:45.797130  8249 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.797552  8141 leader_election.cc:304] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd; no voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.798568  8324 raft_consensus.cc:2749] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:45.801599  8333 leader_election.cc:290] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.802268  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.803702  8248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
W20260709 13:00:45.806535  8067 leader_election.cc:343] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:45.807386  8064 leader_election.cc:343] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.807999  8064 leader_election.cc:304] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbb00b12d0e4da58236f580ae8e2df9; no voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:45.809186  8290 tablet_bootstrap.cc:492] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.809796  8290 ts_tablet_manager.cc:1403] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.070s	user 0.018s	sys 0.006s
I20260709 13:00:45.809986  8333 raft_consensus.cc:2749] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:45.806143  8297 raft_consensus.cc:515] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.812992  8290 raft_consensus.cc:359] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.813342  8249 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 1.
I20260709 13:00:45.813799  8290 raft_consensus.cc:385] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.814189  8290 raft_consensus.cc:740] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.815448  8290 consensus_queue.cc:260] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.815622  8067 leader_election.cc:304] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:45.816807  8333 raft_consensus.cc:2804] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.817422  8333 raft_consensus.cc:697] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:45.789243  8291 raft_consensus.cc:359] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.818146  8291 raft_consensus.cc:385] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.804558  8175 raft_consensus.cc:2468] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 1.
I20260709 13:00:45.818708  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:45.818600  8333 consensus_queue.cc:237] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
W20260709 13:00:45.821069  8141 leader_election.cc:343] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.823014  8213 leader_election.cc:304] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.824105  8310 raft_consensus.cc:2804] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.824213  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:45.824669  8310 raft_consensus.cc:697] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Becoming Leader. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:45.824853  8249 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.825009  8290 ts_tablet_manager.cc:1434] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.001s
I20260709 13:00:45.825634  8310 consensus_queue.cc:237] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.827020  8297 leader_election.cc:290] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.828362  8248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:45.804230  8250 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 1.
W20260709 13:00:45.830054  8142 leader_election.cc:343] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.831310  8067 leader_election.cc:304] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:45.832182  8328 raft_consensus.cc:2804] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.832649  8328 raft_consensus.cc:697] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:45.826233  8290 tablet_bootstrap.cc:492] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.833600  8328 consensus_queue.cc:237] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.835073  8249 raft_consensus.cc:2468] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:45.837167  8142 leader_election.cc:304] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.838558  8297 raft_consensus.cc:2804] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.839231  8297 raft_consensus.cc:697] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:45.840137  8297 consensus_queue.cc:237] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.841583  8101 raft_consensus.cc:2468] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:45.844265  8141 leader_election.cc:304] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: c867ee48f32e4dc1808ce09d00c3c5f2
I20260709 13:00:45.844640  8290 tablet_bootstrap.cc:654] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.845378  8329 raft_consensus.cc:2804] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.845935  8329 raft_consensus.cc:697] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:45.847261  8326 leader_election.cc:290] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.846935  8329 consensus_queue.cc:237] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.818475  8291 raft_consensus.cc:740] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.849047  8291 consensus_queue.cc:260] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.851900  8291 ts_tablet_manager.cc:1434] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.066s	user 0.006s	sys 0.000s
I20260709 13:00:45.856346  8324 raft_consensus.cc:493] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.856938  8324 raft_consensus.cc:515] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.858637  8290 tablet_bootstrap.cc:492] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.859151  8324 leader_election.cc:290] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.859256  8290 ts_tablet_manager.cc:1403] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.033s	user 0.021s	sys 0.000s
I20260709 13:00:45.861382  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
W20260709 13:00:45.863169  8142 leader_election.cc:343] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.863020  8290 raft_consensus.cc:359] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.864346  8290 raft_consensus.cc:385] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.864807  8290 raft_consensus.cc:740] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.863525  7996 catalog_manager.cc:5697] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 1 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.866051  8290 consensus_queue.cc:260] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.869671  8290 ts_tablet_manager.cc:1434] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
I20260709 13:00:45.871944  8290 tablet_bootstrap.cc:492] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.873723  8311 raft_consensus.cc:493] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.874773  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:45.874406  8311 raft_consensus.cc:515] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.875523  8101 raft_consensus.cc:2468] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:45.876770  8141 leader_election.cc:304] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: c867ee48f32e4dc1808ce09d00c3c5f2
I20260709 13:00:45.878407  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:45.878264  8329 raft_consensus.cc:2804] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.879278  8100 raft_consensus.cc:2393] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c867ee48f32e4dc1808ce09d00c3c5f2 in current term 1: Already voted for candidate f9eb3440010246a59b2fe21ddc2736cd in this term.
I20260709 13:00:45.879390  8329 raft_consensus.cc:493] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.879725  8311 leader_election.cc:290] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:45.853175  8291 tablet_bootstrap.cc:492] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:45.879925  8329 raft_consensus.cc:3060] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.883323  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.885504  8213 leader_election.cc:304] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2; no voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:45.887071  8311 raft_consensus.cc:3060] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.895340  7996 catalog_manager.cc:5697] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.904537  8333 raft_consensus.cc:493] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.905097  8333 raft_consensus.cc:515] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.909618  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dff4362b5f14c30ab3505adf6e5e89f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
W20260709 13:00:45.911145  8067 leader_election.cc:343] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.909657  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dff4362b5f14c30ab3505adf6e5e89f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:45.907438  8333 leader_election.cc:290] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.913684  8311 raft_consensus.cc:2749] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:45.914479  8290 tablet_bootstrap.cc:654] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:45.914798  8064 leader_election.cc:343] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.915398  8064 leader_election.cc:304] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbb00b12d0e4da58236f580ae8e2df9; no voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:45.917021  8333 raft_consensus.cc:2749] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:45.931347  7996 catalog_manager.cc:5697] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.936034  7997 catalog_manager.cc:5697] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.937125  8290 tablet_bootstrap.cc:492] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:45.937644  8290 ts_tablet_manager.cc:1403] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.066s	user 0.019s	sys 0.000s
I20260709 13:00:45.939965  8329 raft_consensus.cc:515] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.941315  8293 tablet_bootstrap.cc:492] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.944973  8293 ts_tablet_manager.cc:1403] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.171s	user 0.042s	sys 0.053s
I20260709 13:00:45.947257  8101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:45.947924  8101 raft_consensus.cc:3060] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.948076  8293 raft_consensus.cc:359] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.948859  8293 raft_consensus.cc:385] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.949170  8293 raft_consensus.cc:740] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.949916  8293 consensus_queue.cc:260] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:45.899647  7998 catalog_manager.cc:5697] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 1 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.941817  8290 raft_consensus.cc:359] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.951942  8290 raft_consensus.cc:385] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.952296  8290 raft_consensus.cc:740] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.953011  8290 consensus_queue.cc:260] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.955188  8293 ts_tablet_manager.cc:1434] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.006s
I20260709 13:00:45.955988  8290 ts_tablet_manager.cc:1434] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.002s
I20260709 13:00:45.956211  8293 tablet_bootstrap.cc:492] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.956866  8290 tablet_bootstrap.cc:492] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd: Bootstrap starting.
I20260709 13:00:45.963385  8293 tablet_bootstrap.cc:654] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.955946  8101 raft_consensus.cc:2468] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:45.944326  8291 tablet_bootstrap.cc:654] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.965047  8141 leader_election.cc:304] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:45.966382  8297 raft_consensus.cc:2804] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.966888  8290 tablet_bootstrap.cc:654] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.967648  8297 raft_consensus.cc:697] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:45.968796  8297 consensus_queue.cc:237] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.977070  8293 tablet_bootstrap.cc:492] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:45.977922  8293 ts_tablet_manager.cc:1403] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.001s
I20260709 13:00:45.981594  8293 raft_consensus.cc:359] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.982646  8293 raft_consensus.cc:385] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.982376  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:45.983083  8329 leader_election.cc:290] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:45.983198  8293 raft_consensus.cc:740] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
W20260709 13:00:45.984638  8142 leader_election.cc:343] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.986348  8293 consensus_queue.cc:260] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:45.989166  8293 ts_tablet_manager.cc:1434] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20260709 13:00:45.991729  8293 tablet_bootstrap.cc:492] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:45.999220  8293 tablet_bootstrap.cc:654] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.000358  8290 tablet_bootstrap.cc:492] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd: No bootstrap required, opened a new log
I20260709 13:00:46.000890  8290 ts_tablet_manager.cc:1403] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.019s
I20260709 13:00:45.999897  7996 catalog_manager.cc:5697] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 1 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.001498  8291 tablet_bootstrap.cc:492] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:46.001991  8291 ts_tablet_manager.cc:1403] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.149s	user 0.028s	sys 0.011s
I20260709 13:00:46.003501  8290 raft_consensus.cc:359] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.004195  8290 raft_consensus.cc:385] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.004446  8290 raft_consensus.cc:740] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.005292  8290 consensus_queue.cc:260] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.005924  8291 raft_consensus.cc:359] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.006886  8291 raft_consensus.cc:385] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.007272  8291 raft_consensus.cc:740] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.007925  8290 ts_tablet_manager.cc:1434] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 13:00:46.008560  8291 consensus_queue.cc:260] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.011179  8293 tablet_bootstrap.cc:492] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:46.011461  8291 ts_tablet_manager.cc:1434] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:46.011646  8293 ts_tablet_manager.cc:1403] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.005s
I20260709 13:00:46.012956  8291 tablet_bootstrap.cc:492] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:46.014428  8293 raft_consensus.cc:359] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.015175  8293 raft_consensus.cc:385] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.015473  8293 raft_consensus.cc:740] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.016201  8293 consensus_queue.cc:260] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.018268  8293 ts_tablet_manager.cc:1434] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.001s
I20260709 13:00:46.019236  8293 tablet_bootstrap.cc:492] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: Bootstrap starting.
I20260709 13:00:46.020191  8291 tablet_bootstrap.cc:654] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.025388  8293 tablet_bootstrap.cc:654] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.038599  8291 tablet_bootstrap.cc:492] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:46.038599  8293 tablet_bootstrap.cc:492] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: No bootstrap required, opened a new log
I20260709 13:00:46.039184  8291 ts_tablet_manager.cc:1403] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.026s	user 0.007s	sys 0.015s
I20260709 13:00:46.039192  8293 ts_tablet_manager.cc:1403] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.004s
I20260709 13:00:46.042110  8293 raft_consensus.cc:359] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.042110  8291 raft_consensus.cc:359] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.042879  8293 raft_consensus.cc:385] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.042915  8291 raft_consensus.cc:385] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.043287  8293 raft_consensus.cc:740] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.043318  8291 raft_consensus.cc:740] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.044047  8293 consensus_queue.cc:260] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.044047  8291 consensus_queue.cc:260] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.046183  8291 ts_tablet_manager.cc:1434] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:00:46.046191  8293 ts_tablet_manager.cc:1434] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:46.047380  8291 tablet_bootstrap.cc:492] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:46.053144  8291 tablet_bootstrap.cc:654] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.057873  8291 tablet_bootstrap.cc:492] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:46.058310  8291 ts_tablet_manager.cc:1403] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 13:00:46.060341  8291 raft_consensus.cc:359] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.060771  8291 raft_consensus.cc:385] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.060933  8291 raft_consensus.cc:740] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.061368  8291 consensus_queue.cc:260] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.063136  8291 ts_tablet_manager.cc:1434] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:00:46.064003  8291 tablet_bootstrap.cc:492] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2: Bootstrap starting.
I20260709 13:00:46.068379  8291 tablet_bootstrap.cc:654] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.072502  8291 tablet_bootstrap.cc:492] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2: No bootstrap required, opened a new log
I20260709 13:00:46.072890  8291 ts_tablet_manager.cc:1403] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:46.074918  8291 raft_consensus.cc:359] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.075364  8291 raft_consensus.cc:385] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.075603  8291 raft_consensus.cc:740] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.076152  8291 consensus_queue.cc:260] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.077877  8291 ts_tablet_manager.cc:1434] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.003s
I20260709 13:00:46.079452  8101 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.080497  8297 raft_consensus.cc:493] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.081033  8297 raft_consensus.cc:515] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.083536  8297 leader_election.cc:290] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.084403  8328 raft_consensus.cc:493] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.084066  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dff4362b5f14c30ab3505adf6e5e89f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:46.085204  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dff4362b5f14c30ab3505adf6e5e89f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:46.085328  8100 raft_consensus.cc:2468] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:46.085204  8328 raft_consensus.cc:515] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.086091  8249 raft_consensus.cc:2468] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:46.087009  8141 leader_election.cc:304] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:46.087704  8328 leader_election.cc:290] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.088583  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:46.088879  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:46.089761  8297 raft_consensus.cc:2804] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.089838  8249 raft_consensus.cc:2393] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbbb00b12d0e4da58236f580ae8e2df9 in current term 1: Already voted for candidate f9eb3440010246a59b2fe21ddc2736cd in this term.
I20260709 13:00:46.089895  8310 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:46.090360  8297 raft_consensus.cc:493] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.090893  8297 raft_consensus.cc:3060] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.091255  8067 leader_election.cc:304] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbb00b12d0e4da58236f580ae8e2df9; no voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:46.092125  8328 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.099478  8297 raft_consensus.cc:515] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.101981  8297 leader_election.cc:290] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.105647  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dff4362b5f14c30ab3505adf6e5e89f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:46.105760  8353 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:46.106468  8249 raft_consensus.cc:3060] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.106617  8328 raft_consensus.cc:2749] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.110191  8099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dff4362b5f14c30ab3505adf6e5e89f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:46.111289  8099 raft_consensus.cc:3060] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.126643  8249 raft_consensus.cc:2468] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:46.128028  8142 leader_election.cc:304] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:46.129001  8297 raft_consensus.cc:2804] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:46.129565  8297 raft_consensus.cc:697] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:46.130503  8297 consensus_queue.cc:237] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.131907  8099 raft_consensus.cc:2468] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:46.143422  8328 raft_consensus.cc:493] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.144083  8328 raft_consensus.cc:515] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:46.145196  7999 catalog_manager.cc:5697] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 1 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.146435  8328 leader_election.cc:290] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.147342  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd" is_pre_election: true
I20260709 13:00:46.148204  8175 raft_consensus.cc:2393] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbbb00b12d0e4da58236f580ae8e2df9 in current term 1: Already voted for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in this term.
I20260709 13:00:46.149044  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:46.150578  8067 leader_election.cc:304] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbb00b12d0e4da58236f580ae8e2df9; no voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:46.151455  8328 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.158545  8328 raft_consensus.cc:2749] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.160511  8297 raft_consensus.cc:493] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.160979  8329 consensus_queue.cc:1048] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.161067  8297 raft_consensus.cc:515] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.164206  8099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:46.164796  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:46.164913  8099 raft_consensus.cc:2468] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:46.165386  8249 raft_consensus.cc:2468] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 0.
I20260709 13:00:46.165537  8297 leader_election.cc:290] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.166527  8141 leader_election.cc:304] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:46.167655  8297 raft_consensus.cc:2804] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.167984  8297 raft_consensus.cc:493] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.168232  8297 raft_consensus.cc:3060] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.175474  8297 raft_consensus.cc:515] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.177786  8297 leader_election.cc:290] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.178262  8324 consensus_queue.cc:1048] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.179291  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:46.179881  8249 raft_consensus.cc:3060] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.180042  8099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:46.180743  8099 raft_consensus.cc:3060] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.188614  8249 raft_consensus.cc:2468] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:46.189872  8142 leader_election.cc:304] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:46.190557  8324 raft_consensus.cc:2804] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:46.191180  8324 raft_consensus.cc:697] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:46.192013  8324 consensus_queue.cc:237] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.193749  8099 raft_consensus.cc:2468] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 1.
I20260709 13:00:46.196470  8297 raft_consensus.cc:493] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.196996  8297 raft_consensus.cc:515] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.199564  8297 leader_election.cc:290] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.200398  8099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" is_pre_election: true
I20260709 13:00:46.200985  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" is_pre_election: true
I20260709 13:00:46.201856  8249 raft_consensus.cc:2393] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9eb3440010246a59b2fe21ddc2736cd in current term 1: Already voted for candidate fbbb00b12d0e4da58236f580ae8e2df9 in this term.
I20260709 13:00:46.202522  8328 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.203697  8142 leader_election.cc:304] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd; no voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:46.204842  8326 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.207101  7996 catalog_manager.cc:5697] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 1 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.212839  8326 raft_consensus.cc:2749] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.220836  8363 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:46.234961  8363 consensus_queue.cc:1048] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.246294  8333 consensus_queue.cc:1048] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.257755  8310 consensus_queue.cc:1048] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.268973  8353 consensus_queue.cc:1048] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.299324  8326 consensus_queue.cc:1048] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.308629  8249 raft_consensus.cc:3060] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.316964  8297 consensus_queue.cc:1048] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.360260  8297 consensus_queue.cc:1048] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.369462  8249 raft_consensus.cc:3060] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.377472  8324 consensus_queue.cc:1048] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.387159  8326 consensus_queue.cc:1048] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:46.396668  8329 consensus_queue.cc:1048] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:46.424675  8249 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:36166
I20260709 13:00:46.425210  8249 raft_consensus.cc:606] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:46.431087  8249 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:36166
I20260709 13:00:46.431594  8249 raft_consensus.cc:606] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:46.445271  8175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:51414
I20260709 13:00:46.445917  8175 raft_consensus.cc:606] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:46.451714  8175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:51414
I20260709 13:00:46.452351  8175 raft_consensus.cc:606] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:46.475332  8099 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
mode: GRACEFUL
new_leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
 from {username='slave'} at 127.0.0.1:45948
I20260709 13:00:46.475879  8099 raft_consensus.cc:606] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Received request to transfer leadership to TS f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:46.484220  8175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
mode: GRACEFUL
new_leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:51414
I20260709 13:00:46.484740  8175 raft_consensus.cc:606] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Received request to transfer leadership to TS c867ee48f32e4dc1808ce09d00c3c5f2
I20260709 13:00:46.587163  8329 consensus_queue.cc:1048] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.595479  8329 raft_consensus.cc:993] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd: : Instructing follower c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:46.596060  8329 raft_consensus.cc:1081] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Signalling peer c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:46.598794  8249 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:36150
I20260709 13:00:46.598951  8329 consensus_queue.cc:1048] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.599375  8249 raft_consensus.cc:493] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:46.599766  8249 raft_consensus.cc:3060] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.610392  8249 raft_consensus.cc:515] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.612792  8249 leader_election.cc:290] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 2 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:46.613233  8099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:46.613402  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57723b298b34d6f889d10aa14cd4094" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:46.613867  8099 raft_consensus.cc:3060] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.613950  8175 raft_consensus.cc:3055] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:46.614327  8175 raft_consensus.cc:740] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:46.615162  8175 consensus_queue.cc:260] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.616425  8175 raft_consensus.cc:3060] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.621006  8099 raft_consensus.cc:2468] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 2.
I20260709 13:00:46.622679  8215 leader_election.cc:304] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:46.622907  8175 raft_consensus.cc:2468] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 2.
I20260709 13:00:46.623423  8353 raft_consensus.cc:2804] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:46.623946  8353 raft_consensus.cc:697] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 LEADER]: Becoming Leader. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:46.624751  8353 consensus_queue.cc:237] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.632402  8311 raft_consensus.cc:993] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.632913  8311 raft_consensus.cc:1081] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.632071  7999 catalog_manager.cc:5697] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 reported cstate change: term changed from 1 to 2, leader changed from f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66) to c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:46.634865  8099 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45940
I20260709 13:00:46.635481  8099 raft_consensus.cc:493] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:46.635823  8099 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.642367  8099 raft_consensus.cc:515] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.644176  8099 leader_election.cc:290] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.644905  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:46.645505  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:46.645663  8175 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.646145  8249 raft_consensus.cc:3055] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:46.646481  8249 raft_consensus.cc:740] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:46.647284  8249 consensus_queue.cc:260] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.648476  8249 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.653039  8175 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.654564  8064 leader_election.cc:304] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:46.655018  8249 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.655361  8333 raft_consensus.cc:2804] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:46.655841  8333 raft_consensus.cc:697] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:46.656788  8333 consensus_queue.cc:237] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.662513  8363 raft_consensus.cc:993] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: : Instructing follower f9eb3440010246a59b2fe21ddc2736cd to start an election
I20260709 13:00:46.663002  8333 raft_consensus.cc:1081] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Signalling peer f9eb3440010246a59b2fe21ddc2736cd to start an election
I20260709 13:00:46.664702  8175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
 from {username='slave'} at 127.0.0.1:51398
I20260709 13:00:46.665282  8175 raft_consensus.cc:493] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:46.665728  8175 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.664937  7999 catalog_manager.cc:5697] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 1 to 2, leader changed from c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.671701  8175 raft_consensus.cc:515] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.673775  8175 leader_election.cc:290] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 2 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.674551  8099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:46.674777  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:46.675343  8249 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.675515  8099 raft_consensus.cc:3055] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:46.675868  8099 raft_consensus.cc:740] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:46.676642  8099 consensus_queue.cc:260] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.677886  8099 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.682771  8249 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 2.
I20260709 13:00:46.684093  8142 leader_election.cc:304] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:46.684661  8099 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 2.
I20260709 13:00:46.684940  8329 raft_consensus.cc:2804] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:46.685496  8329 raft_consensus.cc:697] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:46.686426  8329 consensus_queue.cc:237] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.694130  7995 catalog_manager.cc:5697] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 1 to 2, leader changed from fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65) to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 2 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.713124  8329 consensus_queue.cc:1048] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.719403  8324 raft_consensus.cc:993] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.719877  8297 raft_consensus.cc:1081] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.722517  8100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45944
I20260709 13:00:46.724316  8100 raft_consensus.cc:493] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:46.724731  8100 raft_consensus.cc:3060] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.724951  8297 consensus_queue.cc:1048] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.732174  8100 raft_consensus.cc:515] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.737545  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:46.738184  8175 raft_consensus.cc:3055] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:46.738108  8100 leader_election.cc:290] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.738598  8175 raft_consensus.cc:740] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:46.738612  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d843ab104dc4b39a7ef8c0f01dc62a4" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:46.739761  8249 raft_consensus.cc:3060] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.739799  8175 consensus_queue.cc:260] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.741110  8175 raft_consensus.cc:3060] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.747309  8249 raft_consensus.cc:2468] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.748663  8175 raft_consensus.cc:2468] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.748723  8067 leader_election.cc:304] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:46.749615  8363 raft_consensus.cc:2804] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:46.750041  8363 raft_consensus.cc:697] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:46.750873  8363 consensus_queue.cc:237] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.757792  7995 catalog_manager.cc:5697] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 1 to 2, leader changed from f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.834830  8353 raft_consensus.cc:993] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.835356  8383 raft_consensus.cc:1081] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.837077  8100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45940
I20260709 13:00:46.837620  8100 raft_consensus.cc:493] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:46.837970  8100 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.842808  8100 raft_consensus.cc:515] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:46.844522  8100 leader_election.cc:290] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.845624  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:46.845781  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:46.846205  8175 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.846398  8249 raft_consensus.cc:3055] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:46.846788  8249 raft_consensus.cc:740] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:46.847512  8249 consensus_queue.cc:260] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:46.848506  8249 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.851603  8175 raft_consensus.cc:2468] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.852713  8064 leader_election.cc:304] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:46.853591  8363 raft_consensus.cc:2804] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:46.854053  8363 raft_consensus.cc:697] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:46.854573  8249 raft_consensus.cc:2468] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.854842  8363 consensus_queue.cc:237] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:46.862739  8324 raft_consensus.cc:993] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.862141  7995 catalog_manager.cc:5697] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 1 to 2, leader changed from c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:46.863204  8324 raft_consensus.cc:1081] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:46.864965  8100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45944
I20260709 13:00:46.865648  8100 raft_consensus.cc:493] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:46.866053  8100 raft_consensus.cc:3060] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.872117  8100 raft_consensus.cc:515] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.874226  8100 leader_election.cc:290] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:46.875067  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:46.875231  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5d641112e2a4f6ea3dea8fbbc499f64" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:46.875680  8175 raft_consensus.cc:3055] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:46.875818  8249 raft_consensus.cc:3060] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.876117  8175 raft_consensus.cc:740] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:46.876943  8175 consensus_queue.cc:260] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.878084  8175 raft_consensus.cc:3060] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.882058  8249 raft_consensus.cc:2468] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.883533  8067 leader_election.cc:304] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:46.884130  8175 raft_consensus.cc:2468] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:46.884459  8363 raft_consensus.cc:2804] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:46.884986  8363 raft_consensus.cc:697] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:46.885865  8363 consensus_queue.cc:237] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:46.894151  7995 catalog_manager.cc:5697] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 1 to 2, leader changed from f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:47.021040  8100 raft_consensus.cc:1275] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.022667  8383 consensus_queue.cc:1048] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.030588  8175 raft_consensus.cc:1275] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.032004  8311 consensus_queue.cc:1048] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.062901  8175 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.064237  8249 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.064674  8363 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.066270  8363 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.102411  8100 raft_consensus.cc:1275] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Refusing update from remote peer f9eb3440010246a59b2fe21ddc2736cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.103813  8297 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.111562  8249 raft_consensus.cc:1275] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Refusing update from remote peer f9eb3440010246a59b2fe21ddc2736cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.112876  8297 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.141929  8175 raft_consensus.cc:1275] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.143332  8298 consensus_queue.cc:1048] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.151530  8249 raft_consensus.cc:1275] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.153264  8333 consensus_queue.cc:1048] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.302146  8249 raft_consensus.cc:1275] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.303404  8333 consensus_queue.cc:1048] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.312412  8175 raft_consensus.cc:1275] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.314100  8333 consensus_queue.cc:1048] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.420643  8175 raft_consensus.cc:1275] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.422180  8333 consensus_queue.cc:1048] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:47.432103  8249 raft_consensus.cc:1275] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:47.433531  8363 consensus_queue.cc:1048] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:49.521749  7481 auto_leader_rebalancer-test.cc:348] The leader distribution is 
I20260709 13:00:49.522629  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:49.541991  8100 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
mode: GRACEFUL
new_leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:45964
I20260709 13:00:49.542517  8100 raft_consensus.cc:606] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Received request to transfer leadership to TS c867ee48f32e4dc1808ce09d00c3c5f2
I20260709 13:00:49.544781  8100 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
mode: GRACEFUL
new_leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:45964
I20260709 13:00:49.545285  8100 raft_consensus.cc:606] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Received request to transfer leadership to TS c867ee48f32e4dc1808ce09d00c3c5f2
I20260709 13:00:49.546358  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:00:49.546797  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:49.935489  8363 raft_consensus.cc:993] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9: : Instructing follower c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:49.935928  8363 raft_consensus.cc:1081] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Signalling peer c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:49.937361  8249 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:36162
I20260709 13:00:49.937829  8249 raft_consensus.cc:493] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:49.938205  8249 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.942190  8249 raft_consensus.cc:515] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:49.943693  8249 leader_election.cc:290] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 2 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:49.944638  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:49.944896  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:49.945272  8100 raft_consensus.cc:3055] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:49.945430  8175 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.945698  8100 raft_consensus.cc:740] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:49.946563  8100 consensus_queue.cc:260] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:49.947773  8100 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.952303  8175 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 2.
I20260709 13:00:49.953786  8213 leader_election.cc:304] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:49.954308  8100 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 2.
I20260709 13:00:49.954558  8311 raft_consensus.cc:2804] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:49.955621  8311 raft_consensus.cc:697] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 LEADER]: Becoming Leader. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:49.956358  8311 consensus_queue.cc:237] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:49.967011  7999 catalog_manager.cc:5697] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 reported cstate change: term changed from 1 to 2, leader changed from fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65) to c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:49.991256  8392 raft_consensus.cc:993] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: : Instructing follower c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:49.991703  8392 raft_consensus.cc:1081] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Signalling peer c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:49.993587  8249 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:36162
I20260709 13:00:49.994095  8249 raft_consensus.cc:493] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:49.994441  8249 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:49.999639  8249 raft_consensus.cc:515] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.001312  8249 leader_election.cc:290] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 3 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:50.002283  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:50.002358  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:50.002864  8100 raft_consensus.cc:3055] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:50.002921  8175 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.003228  8100 raft_consensus.cc:740] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:50.003907  8100 consensus_queue.cc:260] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.004858  8100 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.007499  8175 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 3.
I20260709 13:00:50.008553  8213 leader_election.cc:304] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:50.009452  8311 raft_consensus.cc:2804] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:50.009985  8311 raft_consensus.cc:697] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 LEADER]: Becoming Leader. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:50.010329  8100 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 3.
I20260709 13:00:50.010958  8311 consensus_queue.cc:237] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.018805  7996 catalog_manager.cc:5697] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 reported cstate change: term changed from 2 to 3, leader changed from fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65) to c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:50.489473  8100 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:50.490828  8311 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:50.502125  8175 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:50.504401  8402 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:50.547564  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:50.560101  8248 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:36166
I20260709 13:00:50.560652  8248 raft_consensus.cc:606] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:50.570760  8248 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:36166
I20260709 13:00:50.571341  8248 raft_consensus.cc:606] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:50.576122  8175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:51414
I20260709 13:00:50.576545  8175 raft_consensus.cc:606] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:50.578979  8100 raft_consensus.cc:1275] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:50.580240  8402 consensus_queue.cc:1048] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:50.582037  8175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:51414
I20260709 13:00:50.582559  8175 raft_consensus.cc:606] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:50.585515  8175 raft_consensus.cc:1275] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:50.587321  8402 consensus_queue.cc:1048] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:50.588604  8393 raft_consensus.cc:993] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.589007  8393 raft_consensus.cc:1081] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.589537  8175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
mode: GRACEFUL
new_leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:51414
I20260709 13:00:50.590086  8175 raft_consensus.cc:606] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 LEADER]: Received request to transfer leadership to TS fbbb00b12d0e4da58236f580ae8e2df9
I20260709 13:00:50.597508  8100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45940
I20260709 13:00:50.598153  8100 raft_consensus.cc:493] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:50.598567  8100 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.605711  8100 raft_consensus.cc:515] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.608441  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:50.609097  8176 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.613900  8100 leader_election.cc:290] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 3 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:50.615172  8248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0d428e367ce4dffa8d66f0deb321f6a" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:50.615691  8248 raft_consensus.cc:3055] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:50.615959  8248 raft_consensus.cc:740] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:50.616081  8176 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 3.
I20260709 13:00:50.616617  8248 consensus_queue.cc:260] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.617086  8064 leader_election.cc:304] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:50.617625  8248 raft_consensus.cc:3060] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.617893  8392 raft_consensus.cc:2804] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:50.619148  8392 raft_consensus.cc:697] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:50.620014  8392 consensus_queue.cc:237] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.624169  8248 raft_consensus.cc:2468] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 3.
I20260709 13:00:50.626783  7995 catalog_manager.cc:5697] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 2 to 3, leader changed from c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.717027  8406 raft_consensus.cc:993] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.717584  8389 raft_consensus.cc:1081] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.719101  8100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45944
I20260709 13:00:50.719682  8100 raft_consensus.cc:493] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:50.719995  8100 raft_consensus.cc:3060] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.724658  8100 raft_consensus.cc:515] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.726186  8100 leader_election.cc:290] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:50.727003  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:50.727324  8248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6210b6863d4e474f9dec78996ab0f808" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:50.727599  8176 raft_consensus.cc:3055] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:50.727860  8248 raft_consensus.cc:3060] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.727914  8176 raft_consensus.cc:740] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:50.728660  8176 consensus_queue.cc:260] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.729631  8176 raft_consensus.cc:3060] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.733506  8248 raft_consensus.cc:2468] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:50.734146  8176 raft_consensus.cc:2468] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:50.734607  8067 leader_election.cc:304] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:50.735293  8392 raft_consensus.cc:2804] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:50.735749  8392 raft_consensus.cc:697] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:50.736428  8392 consensus_queue.cc:237] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.743219  7995 catalog_manager.cc:5697] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 1 to 2, leader changed from f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.919469  8406 raft_consensus.cc:993] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.919986  8389 raft_consensus.cc:1081] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.921463  8100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45944
I20260709 13:00:50.922094  8100 raft_consensus.cc:493] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:50.922473  8100 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.927256  8406 raft_consensus.cc:993] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.927696  8389 raft_consensus.cc:1081] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.927870  8100 raft_consensus.cc:515] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.929149  8099 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45944
I20260709 13:00:50.929677  8100 leader_election.cc:290] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 3 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:50.929724  8099 raft_consensus.cc:493] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:50.930300  8099 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.930677  8248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:50.931125  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:50.931308  8248 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.931720  8176 raft_consensus.cc:3055] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:50.932116  8176 raft_consensus.cc:740] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:50.933054  8176 consensus_queue.cc:260] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.934197  8176 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:50.937057  8099 raft_consensus.cc:515] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.937958  8248 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 3.
I20260709 13:00:50.939211  8099 leader_election.cc:290] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:50.939390  8176 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 3.
I20260709 13:00:50.939340  8067 leader_election.cc:304] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:50.939929  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:50.940420  8392 raft_consensus.cc:2804] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:50.940462  8175 raft_consensus.cc:3055] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:50.940927  8175 raft_consensus.cc:740] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:50.941177  8392 raft_consensus.cc:697] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:50.941531  8248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:50.942106  8248 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.941664  8175 consensus_queue.cc:260] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.941975  8392 consensus_queue.cc:237] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.942996  8175 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.949489  8248 raft_consensus.cc:2468] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:50.949299  7995 catalog_manager.cc:5697] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 2 to 3, leader changed from f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.950109  8175 raft_consensus.cc:2468] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 2.
I20260709 13:00:50.950820  8067 leader_election.cc:304] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:50.951743  8392 raft_consensus.cc:2804] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:50.952267  8392 raft_consensus.cc:697] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:50.952942  8392 consensus_queue.cc:237] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.965011  7995 catalog_manager.cc:5697] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 1 to 2, leader changed from f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.986050  8393 raft_consensus.cc:993] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: : Instructing follower fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.986543  8311 raft_consensus.cc:1081] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 LEADER]: Signalling peer fbbb00b12d0e4da58236f580ae8e2df9 to start an election
I20260709 13:00:50.989279  8099 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
 from {username='slave'} at 127.0.0.1:45940
I20260709 13:00:50.989817  8099 raft_consensus.cc:493] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:50.990139  8099 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:50.995942  8099 raft_consensus.cc:515] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:50.997712  8099 leader_election.cc:290] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 4 election: Requested vote from peers f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:50.998668  8175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:50.998814  8248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:50.999166  8175 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:50.999399  8248 raft_consensus.cc:3055] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:50.999764  8248 raft_consensus.cc:740] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:51.000666  8248 consensus_queue.cc:260] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:51.001767  8248 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:51.003620  8175 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 4.
I20260709 13:00:51.004645  8064 leader_election.cc:304] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9eb3440010246a59b2fe21ddc2736cd, fbbb00b12d0e4da58236f580ae8e2df9; no voters: 
I20260709 13:00:51.005333  8392 raft_consensus.cc:2804] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:51.005895  8392 raft_consensus.cc:697] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 LEADER]: Becoming Leader. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:51.006811  8392 consensus_queue.cc:237] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:51.007671  8248 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbb00b12d0e4da58236f580ae8e2df9 in term 4.
I20260709 13:00:51.013486  7995 catalog_manager.cc:5697] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 reported cstate change: term changed from 3 to 4, leader changed from c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67) to fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65). New cstate: current_term: 4 leader_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:51.150086  8175 raft_consensus.cc:1275] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:51.151324  8392 consensus_queue.cc:1048] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:51.159405  8248 raft_consensus.cc:1275] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:51.161024  8417 consensus_queue.cc:1048] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:51.166895  8249 raft_consensus.cc:1275] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:51.168624  8417 consensus_queue.cc:1048] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:51.177829  8175 raft_consensus.cc:1275] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:51.180131  8409 consensus_queue.cc:1048] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:51.340029  8176 raft_consensus.cc:1275] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:51.341490  8409 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:51.350262  8249 raft_consensus.cc:1275] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:51.351691  8409 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:51.388974  8249 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:51.389317  8248 raft_consensus.cc:1275] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:51.390337  8392 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:51.390686  8409 consensus_queue.cc:1048] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:51.400985  8176 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 4 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:51.402963  8175 raft_consensus.cc:1275] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Refusing update from remote peer fbbb00b12d0e4da58236f580ae8e2df9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:51.404235  8409 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:51.410171  8426 consensus_queue.cc:1048] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:53.615127  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:53.624336  8100 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
mode: GRACEFUL
new_leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:45964
I20260709 13:00:53.624995  8100 raft_consensus.cc:606] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Received request to transfer leadership to TS c867ee48f32e4dc1808ce09d00c3c5f2
I20260709 13:00:53.628007  8100 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
mode: GRACEFUL
new_leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:45964
I20260709 13:00:53.628511  8100 raft_consensus.cc:606] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 LEADER]: Received request to transfer leadership to TS c867ee48f32e4dc1808ce09d00c3c5f2
I20260709 13:00:53.631108  8100 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
mode: GRACEFUL
new_leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
 from {username='slave'} at 127.0.0.1:45964
I20260709 13:00:53.631665  8100 raft_consensus.cc:606] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 LEADER]: Received request to transfer leadership to TS f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:53.634023  8100 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef"
dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
mode: GRACEFUL
new_leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
 from {username='slave'} at 127.0.0.1:45964
I20260709 13:00:53.634578  8100 raft_consensus.cc:606] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Received request to transfer leadership to TS f9eb3440010246a59b2fe21ddc2736cd
I20260709 13:00:53.635787  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:00:53.636330  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:53.687939  8429 raft_consensus.cc:993] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9: : Instructing follower f9eb3440010246a59b2fe21ddc2736cd to start an election
I20260709 13:00:53.688647  8409 raft_consensus.cc:1081] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Signalling peer f9eb3440010246a59b2fe21ddc2736cd to start an election
I20260709 13:00:53.690543  8176 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
 from {username='slave'} at 127.0.0.1:51398
I20260709 13:00:53.691217  8176 raft_consensus.cc:493] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:53.691609  8176 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:53.697214  8176 raft_consensus.cc:515] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:53.699173  8176 leader_election.cc:290] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 3 election: Requested vote from peers c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383), fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213)
I20260709 13:00:53.700443  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:53.700469  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7671f95809843818836fbe6cd9076ef" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:53.701452  8100 raft_consensus.cc:3055] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:53.701520  8249 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:53.701844  8100 raft_consensus.cc:740] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:53.702750  8100 consensus_queue.cc:260] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:53.704182  8100 raft_consensus.cc:3060] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:53.709502  8249 raft_consensus.cc:2468] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 3.
I20260709 13:00:53.711058  8142 leader_election.cc:304] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:53.711620  8100 raft_consensus.cc:2468] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 3.
I20260709 13:00:53.711949  8389 raft_consensus.cc:2804] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:53.713050  8389 raft_consensus.cc:697] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 3 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:53.714089  8389 consensus_queue.cc:237] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } }
I20260709 13:00:53.721803  7996 catalog_manager.cc:5697] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 2 to 3, leader changed from fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65) to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 3 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:53.824695  8409 raft_consensus.cc:993] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: : Instructing follower c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:53.825125  8429 raft_consensus.cc:1081] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 LEADER]: Signalling peer c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:53.826856  8249 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:36162
I20260709 13:00:53.827368  8249 raft_consensus.cc:493] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:53.827672  8249 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:53.832640  8249 raft_consensus.cc:515] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.834354  8249 leader_election.cc:290] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 5 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:53.835251  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:53.835516  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ff8b45bea7f4af481db696cab24dac3" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:53.835894  8100 raft_consensus.cc:3055] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:53.836056  8176 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:53.836264  8100 raft_consensus.cc:740] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:53.837121  8100 consensus_queue.cc:260] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.838459  8100 raft_consensus.cc:3060] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:53.842403  8409 raft_consensus.cc:993] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9: : Instructing follower c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:53.842903  8429 raft_consensus.cc:1081] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Signalling peer c867ee48f32e4dc1808ce09d00c3c5f2 to start an election
I20260709 13:00:53.843675  8176 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 5.
I20260709 13:00:53.844295  8249 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51"
dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
 from {username='slave'} at 127.0.0.1:36162
I20260709 13:00:53.844781  8249 raft_consensus.cc:493] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:53.845146  8249 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:53.845134  8213 leader_election.cc:304] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:53.845739  8100 raft_consensus.cc:2468] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 5.
I20260709 13:00:53.846277  8393 raft_consensus.cc:2804] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:53.847721  8393 raft_consensus.cc:697] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 5 LEADER]: Becoming Leader. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:53.848628  8393 consensus_queue.cc:237] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.850791  8249 raft_consensus.cc:515] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.853430  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:53.853987  8100 raft_consensus.cc:3055] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:53.854288  8100 raft_consensus.cc:740] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:53.854552  8176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a4985df633044ffa4bb78d154489b51" candidate_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
I20260709 13:00:53.855000  8100 consensus_queue.cc:260] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.855201  8176 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:53.854318  8249 leader_election.cc:290] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 3 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305)
I20260709 13:00:53.856035  8100 raft_consensus.cc:3060] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:53.857187  7996 catalog_manager.cc:5697] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 reported cstate change: term changed from 4 to 5, leader changed from fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65) to c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67). New cstate: current_term: 5 leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:53.861634  8176 raft_consensus.cc:2468] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 3.
I20260709 13:00:53.862222  8100 raft_consensus.cc:2468] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c867ee48f32e4dc1808ce09d00c3c5f2 in term 3.
I20260709 13:00:53.862888  8213 leader_election.cc:304] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:53.863855  8393 raft_consensus.cc:2804] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:53.864394  8393 raft_consensus.cc:697] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 LEADER]: Becoming Leader. State: Replica: c867ee48f32e4dc1808ce09d00c3c5f2, State: Running, Role: LEADER
I20260709 13:00:53.865353  8393 consensus_queue.cc:237] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.873099  8409 raft_consensus.cc:993] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: : Instructing follower f9eb3440010246a59b2fe21ddc2736cd to start an election
I20260709 13:00:53.873617  8409 raft_consensus.cc:1081] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 LEADER]: Signalling peer f9eb3440010246a59b2fe21ddc2736cd to start an election
I20260709 13:00:53.875198  8176 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f"
dest_uuid: "f9eb3440010246a59b2fe21ddc2736cd"
 from {username='slave'} at 127.0.0.1:51398
I20260709 13:00:53.875705  8176 raft_consensus.cc:493] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:53.876169  8176 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:53.872936  7996 catalog_manager.cc:5697] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 reported cstate change: term changed from 2 to 3, leader changed from fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65) to c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:53.882104  8176 raft_consensus.cc:515] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.883937  8176 leader_election.cc:290] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 4 election: Requested vote from peers fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213), c867ee48f32e4dc1808ce09d00c3c5f2 (127.7.78.67:37383)
I20260709 13:00:53.884924  8100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fbbb00b12d0e4da58236f580ae8e2df9"
I20260709 13:00:53.885526  8100 raft_consensus.cc:3055] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:53.885723  8249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd7704764cb487ba1975b42e16e384f" candidate_uuid: "f9eb3440010246a59b2fe21ddc2736cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2"
I20260709 13:00:53.886313  8100 raft_consensus.cc:740] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fbbb00b12d0e4da58236f580ae8e2df9, State: Running, Role: LEADER
I20260709 13:00:53.886848  8249 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:53.887274  8100 consensus_queue.cc:260] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.888473  8100 raft_consensus.cc:3060] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:53.893172  8249 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 4.
I20260709 13:00:53.894467  8100 raft_consensus.cc:2468] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eb3440010246a59b2fe21ddc2736cd in term 4.
I20260709 13:00:53.894467  8142 leader_election.cc:304] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c867ee48f32e4dc1808ce09d00c3c5f2, f9eb3440010246a59b2fe21ddc2736cd; no voters: 
I20260709 13:00:53.895543  8389 raft_consensus.cc:2804] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:53.896003  8389 raft_consensus.cc:697] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 4 LEADER]: Becoming Leader. State: Replica: f9eb3440010246a59b2fe21ddc2736cd, State: Running, Role: LEADER
I20260709 13:00:53.896800  8389 consensus_queue.cc:237] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } }
I20260709 13:00:53.903833  7996 catalog_manager.cc:5697] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd reported cstate change: term changed from 3 to 4, leader changed from fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65) to f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66). New cstate: current_term: 4 leader_uuid: "f9eb3440010246a59b2fe21ddc2736cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:54.094455  8100 raft_consensus.cc:1275] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Refusing update from remote peer f9eb3440010246a59b2fe21ddc2736cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:54.095759  8389 consensus_queue.cc:1048] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:54.106948  8249 raft_consensus.cc:1275] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Refusing update from remote peer f9eb3440010246a59b2fe21ddc2736cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:54.108621  8442 consensus_queue.cc:1048] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:54.349779  8175 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 5 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:00:54.350852  8100 raft_consensus.cc:1275] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:54.351253  8436 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:54.352044  8393 consensus_queue.cc:1048] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:54.362341  8176 raft_consensus.cc:1275] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:54.365720  8248 raft_consensus.cc:1275] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Refusing update from remote peer f9eb3440010246a59b2fe21ddc2736cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:54.367630  8442 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c867ee48f32e4dc1808ce09d00c3c5f2" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:54.368294  8436 consensus_queue.cc:1048] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eb3440010246a59b2fe21ddc2736cd" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:54.388513  8100 raft_consensus.cc:1275] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 5 FOLLOWER]: Refusing update from remote peer c867ee48f32e4dc1808ce09d00c3c5f2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:00:54.390985  8436 consensus_queue.cc:1048] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:54.396834  8099 raft_consensus.cc:1275] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 FOLLOWER]: Refusing update from remote peer f9eb3440010246a59b2fe21ddc2736cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:54.400821  8389 consensus_queue.cc:1048] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbb00b12d0e4da58236f580ae8e2df9" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:54.637187  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:54.643108  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
I20260709 13:00:54.666836  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:54.667596  7481 tablet_replica.cc:333] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.668637  7481 raft_consensus.cc:2243] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.669061  7481 raft_consensus.cc:2272] T 5ff8b45bea7f4af481db696cab24dac3 P fbbb00b12d0e4da58236f580ae8e2df9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.671416  7481 tablet_replica.cc:333] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.671880  7481 raft_consensus.cc:2243] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.672238  7481 raft_consensus.cc:2272] T 0a4985df633044ffa4bb78d154489b51 P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.674474  7481 tablet_replica.cc:333] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.674999  7481 raft_consensus.cc:2243] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.675714  7481 raft_consensus.cc:2272] T a5d641112e2a4f6ea3dea8fbbc499f64 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.677608  7481 tablet_replica.cc:333] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.678088  7481 raft_consensus.cc:2243] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.678473  7481 raft_consensus.cc:2272] T d7671f95809843818836fbe6cd9076ef P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.680637  7481 tablet_replica.cc:333] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.681200  7481 raft_consensus.cc:2243] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.681666  7481 raft_consensus.cc:2272] T 7dff4362b5f14c30ab3505adf6e5e89f P fbbb00b12d0e4da58236f580ae8e2df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.683440  7481 tablet_replica.cc:333] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.683895  7481 raft_consensus.cc:2243] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.684607  7481 raft_consensus.cc:2272] T f0d428e367ce4dffa8d66f0deb321f6a P fbbb00b12d0e4da58236f580ae8e2df9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.686445  7481 tablet_replica.cc:333] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.686889  7481 raft_consensus.cc:2243] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.687233  7481 raft_consensus.cc:2272] T d57723b298b34d6f889d10aa14cd4094 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.688803  7481 tablet_replica.cc:333] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.689280  7481 raft_consensus.cc:2243] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.690568  7481 raft_consensus.cc:2272] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.692668  7481 tablet_replica.cc:333] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.693141  7481 raft_consensus.cc:2243] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.693564  7481 raft_consensus.cc:2272] T bcd7704764cb487ba1975b42e16e384f P fbbb00b12d0e4da58236f580ae8e2df9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.695626  7481 tablet_replica.cc:333] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9: stopping tablet replica
I20260709 13:00:54.696048  7481 raft_consensus.cc:2243] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.696715  7481 raft_consensus.cc:2272] T 6210b6863d4e474f9dec78996ab0f808 P fbbb00b12d0e4da58236f580ae8e2df9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.726913  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
I20260709 13:00:54.745884  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
I20260709 13:00:54.771677  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:54.772456  7481 tablet_replica.cc:333] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.773106  7481 raft_consensus.cc:2243] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.773931  7481 raft_consensus.cc:2272] T bcd7704764cb487ba1975b42e16e384f P f9eb3440010246a59b2fe21ddc2736cd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.776254  7481 tablet_replica.cc:333] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.776789  7481 raft_consensus.cc:2243] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.777235  7481 raft_consensus.cc:2272] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.779175  7481 tablet_replica.cc:333] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.780018  7481 raft_consensus.cc:2243] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.780632  7481 raft_consensus.cc:2272] T 6210b6863d4e474f9dec78996ab0f808 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.783164  7481 tablet_replica.cc:333] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.783910  7481 raft_consensus.cc:2243] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.784998  7481 raft_consensus.cc:2272] T 7dff4362b5f14c30ab3505adf6e5e89f P f9eb3440010246a59b2fe21ddc2736cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.787278  7481 tablet_replica.cc:333] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.787817  7481 raft_consensus.cc:2243] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.788511  7481 raft_consensus.cc:2272] T d7671f95809843818836fbe6cd9076ef P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.790822  7481 tablet_replica.cc:333] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.791343  7481 raft_consensus.cc:2243] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.791787  7481 raft_consensus.cc:2272] T d57723b298b34d6f889d10aa14cd4094 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.793583  7481 tablet_replica.cc:333] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.794064  7481 raft_consensus.cc:2243] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.794528  7481 raft_consensus.cc:2272] T f0d428e367ce4dffa8d66f0deb321f6a P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.796177  7481 tablet_replica.cc:333] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.796667  7481 raft_consensus.cc:2243] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.797102  7481 raft_consensus.cc:2272] T 5ff8b45bea7f4af481db696cab24dac3 P f9eb3440010246a59b2fe21ddc2736cd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.799165  7481 tablet_replica.cc:333] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.799664  7481 raft_consensus.cc:2243] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.800210  7481 raft_consensus.cc:2272] T 0a4985df633044ffa4bb78d154489b51 P f9eb3440010246a59b2fe21ddc2736cd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.802350  7481 tablet_replica.cc:333] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd: stopping tablet replica
I20260709 13:00:54.802816  7481 raft_consensus.cc:2243] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.803179  7481 raft_consensus.cc:2272] T a5d641112e2a4f6ea3dea8fbbc499f64 P f9eb3440010246a59b2fe21ddc2736cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.833343  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
I20260709 13:00:54.852495  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
W20260709 13:00:54.859071  8215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.78.65:33213: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260709 13:00:54.868558  8215 consensus_peers.cc:597] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 -> Peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Couldn't send request to peer fbbb00b12d0e4da58236f580ae8e2df9. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:33213: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:00:54.869086  8215 consensus_peers.cc:597] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 -> Peer fbbb00b12d0e4da58236f580ae8e2df9 (127.7.78.65:33213): Couldn't send request to peer fbbb00b12d0e4da58236f580ae8e2df9. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:33213: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:00:54.883056  8213 consensus_peers.cc:597] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 -> Peer f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305): Couldn't send request to peer f9eb3440010246a59b2fe21ddc2736cd. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:34305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:54.885879  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:54.886466  7481 tablet_replica.cc:333] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.887085  7481 raft_consensus.cc:2243] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.887668  7481 raft_consensus.cc:2272] T a5d641112e2a4f6ea3dea8fbbc499f64 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.889936  7481 tablet_replica.cc:333] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.890407  7481 raft_consensus.cc:2243] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.890815  7481 raft_consensus.cc:2272] T 6210b6863d4e474f9dec78996ab0f808 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.892447  7481 tablet_replica.cc:333] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.892900  7481 raft_consensus.cc:2243] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.893344  7481 raft_consensus.cc:2272] T 7dff4362b5f14c30ab3505adf6e5e89f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.895516  7481 tablet_replica.cc:333] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.896169  7481 raft_consensus.cc:2243] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.897182  7481 raft_consensus.cc:2272] T d57723b298b34d6f889d10aa14cd4094 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:00:54.898208  8213 consensus_peers.cc:597] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 -> Peer f9eb3440010246a59b2fe21ddc2736cd (127.7.78.66:34305): Couldn't send request to peer f9eb3440010246a59b2fe21ddc2736cd. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:34305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:54.899238  7481 tablet_replica.cc:333] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.899729  7481 raft_consensus.cc:2243] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.900081  7481 raft_consensus.cc:2272] T f0d428e367ce4dffa8d66f0deb321f6a P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.901996  7481 tablet_replica.cc:333] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.902464  7481 raft_consensus.cc:2243] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.902824  7481 raft_consensus.cc:2272] T d7671f95809843818836fbe6cd9076ef P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.904836  7481 tablet_replica.cc:333] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.905306  7481 raft_consensus.cc:2243] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.905771  7481 raft_consensus.cc:2272] T 7d843ab104dc4b39a7ef8c0f01dc62a4 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.907398  7481 tablet_replica.cc:333] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.907833  7481 raft_consensus.cc:2243] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:54.908216  7481 raft_consensus.cc:2272] T bcd7704764cb487ba1975b42e16e384f P c867ee48f32e4dc1808ce09d00c3c5f2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.910344  7481 tablet_replica.cc:333] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.911031  7481 raft_consensus.cc:2243] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.911752  7481 raft_consensus.cc:2272] T 0a4985df633044ffa4bb78d154489b51 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.913687  7481 tablet_replica.cc:333] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2: stopping tablet replica
I20260709 13:00:54.914080  7481 raft_consensus.cc:2243] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.914824  7481 raft_consensus.cc:2272] T 5ff8b45bea7f4af481db696cab24dac3 P c867ee48f32e4dc1808ce09d00c3c5f2 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.943992  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
I20260709 13:00:54.962836  7481 master.cc:562] Master@127.7.78.126:39041 shutting down...
I20260709 13:00:54.984090  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:54.984634  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:54.984975  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P 879881d88c0d4f29b35d2c1cad7a8fd9: stopping tablet replica
I20260709 13:00:55.005149  7481 master.cc:584] Master@127.7.78.126:39041 shutdown complete.
c867ee48f32e4dc1808ce09d00c3c5f2  1
f9eb3440010246a59b2fe21ddc2736cd  4
fbbb00b12d0e4da58236f580ae8e2df9  5
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (10723 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 13:00:55.047148  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:44983
I20260709 13:00:55.048310  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:55.054569  8457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:55.054585  8458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:55.054997  8460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:55.055706  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:55.056604  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:55.056807  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:55.056964  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602055056948 us; error 0 us; skew 500 ppm
I20260709 13:00:55.057632  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:55.060221  7481 webserver.cc:533] Webserver started at http://127.7.78.126:43643/ using document root <none> and password file <none>
I20260709 13:00:55.060755  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:55.060950  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:55.061246  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:55.062636  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.067258  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 13:00:55.070999  8465 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.071889  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I20260709 13:00:55.072255  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.072762  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:55.088100  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:55.089601  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:55.132370  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:44983
I20260709 13:00:55.132534  8516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:44983 every 8 connection(s)
I20260709 13:00:55.136736  8517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.148275  8517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Bootstrap starting.
I20260709 13:00:55.153323  8517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:55.158113  8517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: No bootstrap required, opened a new log
I20260709 13:00:55.160216  8517 raft_consensus.cc:359] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" member_type: VOTER }
I20260709 13:00:55.160605  8517 raft_consensus.cc:385] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:55.160786  8517 raft_consensus.cc:740] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a6e2bdaa2043fba12c0ef60545b7fb, State: Initialized, Role: FOLLOWER
I20260709 13:00:55.161278  8517 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" member_type: VOTER }
I20260709 13:00:55.161777  8517 raft_consensus.cc:399] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:00:55.161978  8517 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:00:55.162191  8517 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:55.166597  8517 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" member_type: VOTER }
I20260709 13:00:55.167171  8517 leader_election.cc:304] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2a6e2bdaa2043fba12c0ef60545b7fb; no voters: 
I20260709 13:00:55.168241  8517 leader_election.cc:290] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:00:55.168608  8520 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:55.169870  8520 raft_consensus.cc:697] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 1 LEADER]: Becoming Leader. State: Replica: b2a6e2bdaa2043fba12c0ef60545b7fb, State: Running, Role: LEADER
I20260709 13:00:55.170555  8520 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" member_type: VOTER }
I20260709 13:00:55.171226  8517 sys_catalog.cc:565] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:00:55.175073  8522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2a6e2bdaa2043fba12c0ef60545b7fb. Latest consensus state: current_term: 1 leader_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" member_type: VOTER } }
I20260709 13:00:55.175776  8522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:55.175163  8521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6e2bdaa2043fba12c0ef60545b7fb" member_type: VOTER } }
I20260709 13:00:55.176860  8521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:55.179143  8527 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:00:55.187217  8527 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:00:55.188270  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:00:55.196267  8527 catalog_manager.cc:1383] Generated new cluster ID: 28b95a44a7a04c95a6304baaf904b52f
I20260709 13:00:55.196611  8527 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:00:55.216223  8527 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:00:55.217751  8527 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:00:55.231676  8527 catalog_manager.cc:6070] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Generated new TSK 0
I20260709 13:00:55.232335  8527 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:00:55.254705  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:55.260812  8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:55.261880  8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:55.264978  7481 server_base.cc:1061] running on GCE node
W20260709 13:00:55.265072  8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:55.266649  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:55.266952  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:55.267167  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602055267147 us; error 0 us; skew 500 ppm
I20260709 13:00:55.268060  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:55.271996  7481 webserver.cc:533] Webserver started at http://127.7.78.65:35167/ using document root <none> and password file <none>
I20260709 13:00:55.272533  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:55.272783  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:55.273257  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:55.275123  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "a0d703ee18464475a290425976246555"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.282559  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.004s
I20260709 13:00:55.287187  8546 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.288163  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:00:55.288523  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "a0d703ee18464475a290425976246555"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.288820  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:55.314630  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:55.316177  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:55.318069  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:55.320511  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:55.320737  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.321026  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:55.321241  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.362772  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:46779
I20260709 13:00:55.362880  8608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:46779 every 8 connection(s)
I20260709 13:00:55.367998  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:55.375392  8613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:55.376277  8614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:55.379297  8616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:55.380406  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:55.381623  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:55.381872  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:55.382071  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602055382052 us; error 0 us; skew 500 ppm
I20260709 13:00:55.382776  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:55.383353  8609 heartbeater.cc:344] Connected to a master server at 127.7.78.126:44983
I20260709 13:00:55.383764  8609 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:55.384588  8609 heartbeater.cc:507] Master 127.7.78.126:44983 requested a full tablet report, sending...
I20260709 13:00:55.385335  7481 webserver.cc:533] Webserver started at http://127.7.78.66:44163/ using document root <none> and password file <none>
I20260709 13:00:55.385892  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:55.386101  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:55.386390  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:55.386946  8482 ts_manager.cc:194] Registered new tserver with Master: a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:00:55.387952  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "d0a2b9c938d94a4fa258e068ea054d84"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.389348  8482 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38096
I20260709 13:00:55.393184  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:00:55.396477  8621 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.397241  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:00:55.397634  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "d0a2b9c938d94a4fa258e068ea054d84"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.397940  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:55.415359  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:55.416589  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:55.418118  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:55.420324  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:55.420568  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.420811  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:55.420989  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.461385  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:39929
I20260709 13:00:55.461529  8683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:39929 every 8 connection(s)
I20260709 13:00:55.466166  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:55.474102  8687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:55.475924  8688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:55.478147  8690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:55.478252  7481 server_base.cc:1061] running on GCE node
I20260709 13:00:55.479683  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:55.479966  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:55.480186  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602055480165 us; error 0 us; skew 500 ppm
I20260709 13:00:55.480957  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:55.481006  8684 heartbeater.cc:344] Connected to a master server at 127.7.78.126:44983
I20260709 13:00:55.481588  8684 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:55.482643  8684 heartbeater.cc:507] Master 127.7.78.126:44983 requested a full tablet report, sending...
I20260709 13:00:55.484007  7481 webserver.cc:533] Webserver started at http://127.7.78.67:41657/ using document root <none> and password file <none>
I20260709 13:00:55.484648  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:55.484827  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:55.484911  8482 ts_manager.cc:194] Registered new tserver with Master: d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:55.485080  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:55.486497  8482 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38104
I20260709 13:00:55.486553  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "1dc41db601fe4b91aa6d461b504da0e7"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.492038  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:00:55.495848  8695 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.496768  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260709 13:00:55.497102  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "1dc41db601fe4b91aa6d461b504da0e7"
format_stamp: "Formatted at 2026-07-09 13:00:55 on dist-test-slave-50cs"
I20260709 13:00:55.497399  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:55.532850  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:55.534143  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:55.535653  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:55.537947  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:55.538125  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.538405  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:55.538579  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:55.579535  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:37975
I20260709 13:00:55.579643  8757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:37975 every 8 connection(s)
I20260709 13:00:55.594595  8758 heartbeater.cc:344] Connected to a master server at 127.7.78.126:44983
I20260709 13:00:55.595012  8758 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:55.595780  8758 heartbeater.cc:507] Master 127.7.78.126:44983 requested a full tablet report, sending...
I20260709 13:00:55.597829  8482 ts_manager.cc:194] Registered new tserver with Master: 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:55.597975  7481 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01513278s
I20260709 13:00:55.599502  8482 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38106
I20260709 13:00:55.625932  8482 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38116:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:00:55.631734  8482 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:00:55.838809  8574 tablet_service.cc:1511] Processing CreateTablet for tablet a346f8961f774e87a3c75fa7e2053cb0 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:00:55.840463  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a346f8961f774e87a3c75fa7e2053cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.841670  8648 tablet_service.cc:1511] Processing CreateTablet for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:00:55.842584  8646 tablet_service.cc:1511] Processing CreateTablet for tablet b387b386886048c8b66e5f499f02aab2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:00:55.842677  8573 tablet_service.cc:1511] Processing CreateTablet for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:00:55.842933  8723 tablet_service.cc:1511] Processing CreateTablet for tablet a346f8961f774e87a3c75fa7e2053cb0 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:00:55.843434  8648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.844106  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b387b386886048c8b66e5f499f02aab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.844233  8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.845640  8722 tablet_service.cc:1511] Processing CreateTablet for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:00:55.846935  8647 tablet_service.cc:1511] Processing CreateTablet for tablet 5559db905acf4ba0b48e9e3e87f49c1c (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:00:55.848412  8647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5559db905acf4ba0b48e9e3e87f49c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.848217  8721 tablet_service.cc:1511] Processing CreateTablet for tablet 5559db905acf4ba0b48e9e3e87f49c1c (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:00:55.849968  8571 tablet_service.cc:1511] Processing CreateTablet for tablet b387b386886048c8b66e5f499f02aab2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:00:55.850730  8645 tablet_service.cc:1511] Processing CreateTablet for tablet eed0f0ed87f74c1fb8320d38bceeac72 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:00:55.850775  8720 tablet_service.cc:1511] Processing CreateTablet for tablet b387b386886048c8b66e5f499f02aab2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:00:55.851470  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b387b386886048c8b66e5f499f02aab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.852474  8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed0f0ed87f74c1fb8320d38bceeac72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.852672  8572 tablet_service.cc:1511] Processing CreateTablet for tablet 5559db905acf4ba0b48e9e3e87f49c1c (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:00:55.847955  8570 tablet_service.cc:1511] Processing CreateTablet for tablet eed0f0ed87f74c1fb8320d38bceeac72 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:00:55.853485  8719 tablet_service.cc:1511] Processing CreateTablet for tablet eed0f0ed87f74c1fb8320d38bceeac72 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:00:55.854246  8572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5559db905acf4ba0b48e9e3e87f49c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.854975  8570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed0f0ed87f74c1fb8320d38bceeac72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.844233  8723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a346f8961f774e87a3c75fa7e2053cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.855995  8649 tablet_service.cc:1511] Processing CreateTablet for tablet a346f8961f774e87a3c75fa7e2053cb0 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:00:55.857548  8721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5559db905acf4ba0b48e9e3e87f49c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.861828  8649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a346f8961f774e87a3c75fa7e2053cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.866818  8719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed0f0ed87f74c1fb8320d38bceeac72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.869489  8720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b387b386886048c8b66e5f499f02aab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.877815  8722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.882009  8569 tablet_service.cc:1511] Processing CreateTablet for tablet b063565a0d8244cabf980570d2fc8a38 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:00:55.891144  8569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b063565a0d8244cabf980570d2fc8a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.903072  8644 tablet_service.cc:1511] Processing CreateTablet for tablet b063565a0d8244cabf980570d2fc8a38 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:00:55.905071  8644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b063565a0d8244cabf980570d2fc8a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.910888  8567 tablet_service.cc:1511] Processing CreateTablet for tablet eeef0c4282c74b3692ab849adb5ce58e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:00:55.912846  8567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeef0c4282c74b3692ab849adb5ce58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.918419  8643 tablet_service.cc:1511] Processing CreateTablet for tablet 3a6a2506bc27408a9b9113a0fa1f069f (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:00:55.923012  8641 tablet_service.cc:1511] Processing CreateTablet for tablet 78613f64484447c487fa71abda1ec1ac (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:00:55.926337  8640 tablet_service.cc:1511] Processing CreateTablet for tablet d1f77a3af1d2472a857b0f7a57069d60 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:00:55.931393  8718 tablet_service.cc:1511] Processing CreateTablet for tablet b063565a0d8244cabf980570d2fc8a38 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:00:55.932924  8718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b063565a0d8244cabf980570d2fc8a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.932453  8566 tablet_service.cc:1511] Processing CreateTablet for tablet 78613f64484447c487fa71abda1ec1ac (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:00:55.934875  8565 tablet_service.cc:1511] Processing CreateTablet for tablet d1f77a3af1d2472a857b0f7a57069d60 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:00:55.935634  8717 tablet_service.cc:1511] Processing CreateTablet for tablet 3a6a2506bc27408a9b9113a0fa1f069f (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:00:55.938684  8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f77a3af1d2472a857b0f7a57069d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.939435  8717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6a2506bc27408a9b9113a0fa1f069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.941602  8568 tablet_service.cc:1511] Processing CreateTablet for tablet 3a6a2506bc27408a9b9113a0fa1f069f (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:00:55.941856  8642 tablet_service.cc:1511] Processing CreateTablet for tablet eeef0c4282c74b3692ab849adb5ce58e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:00:55.948845  8566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78613f64484447c487fa71abda1ec1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.962515  8641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78613f64484447c487fa71abda1ec1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.965847  8643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6a2506bc27408a9b9113a0fa1f069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.968348  8642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeef0c4282c74b3692ab849adb5ce58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.969547  8572 tablet_service.cc:1511] Processing CreateTablet for tablet f057517aa4a1496ab40134759e3d8b54 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:00:55.971572  8646 tablet_service.cc:1511] Processing CreateTablet for tablet f057517aa4a1496ab40134759e3d8b54 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:00:55.972831  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f057517aa4a1496ab40134759e3d8b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.975754  8648 tablet_service.cc:1511] Processing CreateTablet for tablet 8ffc6f04aee94f37b24daec463e6eb45 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:00:55.977089  8648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffc6f04aee94f37b24daec463e6eb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.982192  8773 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:55.988258  8716 tablet_service.cc:1511] Processing CreateTablet for tablet eeef0c4282c74b3692ab849adb5ce58e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:00:55.989748  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeef0c4282c74b3692ab849adb5ce58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.990747  8572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f057517aa4a1496ab40134759e3d8b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.990885  8714 tablet_service.cc:1511] Processing CreateTablet for tablet d1f77a3af1d2472a857b0f7a57069d60 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:00:55.992254  8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f77a3af1d2472a857b0f7a57069d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:55.994473  8715 tablet_service.cc:1511] Processing CreateTablet for tablet 78613f64484447c487fa71abda1ec1ac (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:00:56.001011  8715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78613f64484447c487fa71abda1ec1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.001667  8723 tablet_service.cc:1511] Processing CreateTablet for tablet f057517aa4a1496ab40134759e3d8b54 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:00:56.002954  8723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f057517aa4a1496ab40134759e3d8b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.014061  8640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f77a3af1d2472a857b0f7a57069d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.014531  8646 tablet_service.cc:1511] Processing CreateTablet for tablet 689ad8326d634dee8402f1352a76f0e2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:00:56.015828  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689ad8326d634dee8402f1352a76f0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.024461  8647 tablet_service.cc:1511] Processing CreateTablet for tablet c6ad38ee7735403281a14a504308a8ea (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:00:56.025825  8647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ad38ee7735403281a14a504308a8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.028371  8571 tablet_service.cc:1511] Processing CreateTablet for tablet 8ffc6f04aee94f37b24daec463e6eb45 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:00:56.032300  8645 tablet_service.cc:1511] Processing CreateTablet for tablet 03f3a12327224113b1c5bf090e280a53 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:00:56.033700  8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f3a12327224113b1c5bf090e280a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.036257  8568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6a2506bc27408a9b9113a0fa1f069f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.044816  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffc6f04aee94f37b24daec463e6eb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.045568  8775 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:56.074136  8774 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:56.075644  8573 tablet_service.cc:1511] Processing CreateTablet for tablet 03f3a12327224113b1c5bf090e280a53 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:00:56.077045  8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f3a12327224113b1c5bf090e280a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.082630  8717 tablet_service.cc:1511] Processing CreateTablet for tablet 03f3a12327224113b1c5bf090e280a53 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:00:56.083948  8717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f3a12327224113b1c5bf090e280a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.084484  8722 tablet_service.cc:1511] Processing CreateTablet for tablet 266236cfcd824409b757505321b36d81 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:00:56.085953  8722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266236cfcd824409b757505321b36d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.090566  8720 tablet_service.cc:1511] Processing CreateTablet for tablet 0337bfcbeb3b4e15b96669c8d3ca4692 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:00:56.091697  8720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0337bfcbeb3b4e15b96669c8d3ca4692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.097203  8646 tablet_service.cc:1511] Processing CreateTablet for tablet 266236cfcd824409b757505321b36d81 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:00:56.099912  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266236cfcd824409b757505321b36d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.103588  8642 tablet_service.cc:1511] Processing CreateTablet for tablet 0337bfcbeb3b4e15b96669c8d3ca4692 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:00:56.105055  8642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0337bfcbeb3b4e15b96669c8d3ca4692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.105906  8643 tablet_service.cc:1511] Processing CreateTablet for tablet 4b06c2e2a7c644478ef688b47c11a07a (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:00:56.107379  8643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b06c2e2a7c644478ef688b47c11a07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.112138  8641 tablet_service.cc:1511] Processing CreateTablet for tablet 6b93c5bfc29f4210b1fa8340091624eb (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:00:56.113265  8716 tablet_service.cc:1511] Processing CreateTablet for tablet 4b06c2e2a7c644478ef688b47c11a07a (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:00:56.081005  8569 tablet_service.cc:1511] Processing CreateTablet for tablet 266236cfcd824409b757505321b36d81 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:00:56.115432  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b06c2e2a7c644478ef688b47c11a07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.116008  8569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266236cfcd824409b757505321b36d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.105986  8649 tablet_service.cc:1511] Processing CreateTablet for tablet 162db4080f654c9fbd8c23faf68f6df3 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:00:56.122720  8570 tablet_service.cc:1511] Processing CreateTablet for tablet 0337bfcbeb3b4e15b96669c8d3ca4692 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:00:56.124063  8570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0337bfcbeb3b4e15b96669c8d3ca4692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.130002  8571 tablet_service.cc:1511] Processing CreateTablet for tablet 4b06c2e2a7c644478ef688b47c11a07a (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:00:56.130237  8572 tablet_service.cc:1511] Processing CreateTablet for tablet c6ad38ee7735403281a14a504308a8ea (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:00:56.131448  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b06c2e2a7c644478ef688b47c11a07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.132706  8565 tablet_service.cc:1511] Processing CreateTablet for tablet 162db4080f654c9fbd8c23faf68f6df3 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:00:56.134598  8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162db4080f654c9fbd8c23faf68f6df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.104393  8721 tablet_service.cc:1511] Processing CreateTablet for tablet 8ffc6f04aee94f37b24daec463e6eb45 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:00:56.144371  8721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffc6f04aee94f37b24daec463e6eb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.159461  8715 tablet_service.cc:1511] Processing CreateTablet for tablet 689ad8326d634dee8402f1352a76f0e2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:00:56.160822  8715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689ad8326d634dee8402f1352a76f0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.160877  8716 tablet_service.cc:1511] Processing CreateTablet for tablet c6ad38ee7735403281a14a504308a8ea (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:00:56.162737  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ad38ee7735403281a14a504308a8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.170527  8565 tablet_service.cc:1511] Processing CreateTablet for tablet 6b93c5bfc29f4210b1fa8340091624eb (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:00:56.171180  8573 tablet_service.cc:1511] Processing CreateTablet for tablet 121ce91dbce74f019e528db7b7062a86 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:00:56.172014  8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b93c5bfc29f4210b1fa8340091624eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.177505  8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121ce91dbce74f019e528db7b7062a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.180126  8567 tablet_service.cc:1511] Processing CreateTablet for tablet ea2142382a5d4439915e07c70837c688 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
W20260709 13:00:56.183136  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:56.180331  8571 tablet_service.cc:1511] Processing CreateTablet for tablet a7c87eef075e49ba98e563e6fcc45d83 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:00:56.185627  8566 tablet_service.cc:1511] Processing CreateTablet for tablet bab857d69b5f4bc480340a9324923bc4 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:00:56.189924  8773 tablet_bootstrap.cc:654] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:56.190580  8574 tablet_service.cc:1511] Processing CreateTablet for tablet 689ad8326d634dee8402f1352a76f0e2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:00:56.191797  8647 tablet_service.cc:1511] Processing CreateTablet for tablet 121ce91dbce74f019e528db7b7062a86 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:00:56.193742  8644 tablet_service.cc:1511] Processing CreateTablet for tablet a7c87eef075e49ba98e563e6fcc45d83 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:00:56.195665  8646 tablet_service.cc:1511] Processing CreateTablet for tablet bab857d69b5f4bc480340a9324923bc4 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:00:56.195665  8648 tablet_service.cc:1511] Processing CreateTablet for tablet ea2142382a5d4439915e07c70837c688 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:00:56.203105  8719 tablet_service.cc:1511] Processing CreateTablet for tablet 6b93c5bfc29f4210b1fa8340091624eb (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:00:56.205125  8565 tablet_service.cc:1511] Processing CreateTablet for tablet 2947646ebfc34bd7a5a55312f5812ee2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:00:56.207342  8723 tablet_service.cc:1511] Processing CreateTablet for tablet 121ce91dbce74f019e528db7b7062a86 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:00:56.207978  8641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b93c5bfc29f4210b1fa8340091624eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.230563  8573 tablet_service.cc:1511] Processing CreateTablet for tablet bf63362bb9b4409299528319b7472b04 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:00:56.231185  8644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c87eef075e49ba98e563e6fcc45d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.231870  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab857d69b5f4bc480340a9324923bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.232838  8774 tablet_bootstrap.cc:654] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:56.234576  8647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121ce91dbce74f019e528db7b7062a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.216310  8716 tablet_service.cc:1511] Processing CreateTablet for tablet 162db4080f654c9fbd8c23faf68f6df3 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:00:56.218492  8640 tablet_service.cc:1511] Processing CreateTablet for tablet 2947646ebfc34bd7a5a55312f5812ee2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:00:56.239054  8640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947646ebfc34bd7a5a55312f5812ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.240077  8718 tablet_service.cc:1511] Processing CreateTablet for tablet ea2142382a5d4439915e07c70837c688 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:00:56.241281  8567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2142382a5d4439915e07c70837c688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.241676  8649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162db4080f654c9fbd8c23faf68f6df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.242013  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c87eef075e49ba98e563e6fcc45d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.242632  8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf63362bb9b4409299528319b7472b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.245041  8648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2142382a5d4439915e07c70837c688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.249681  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689ad8326d634dee8402f1352a76f0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.221885  8775 tablet_bootstrap.cc:654] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:56.229454  8719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b93c5bfc29f4210b1fa8340091624eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.253719  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162db4080f654c9fbd8c23faf68f6df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.257733  8723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121ce91dbce74f019e528db7b7062a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.230306  8714 tablet_service.cc:1511] Processing CreateTablet for tablet a7c87eef075e49ba98e563e6fcc45d83 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:00:56.263517  8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c87eef075e49ba98e563e6fcc45d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.265700  8566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab857d69b5f4bc480340a9324923bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.270072  8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947646ebfc34bd7a5a55312f5812ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.271436  8773 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:56.271894  8773 ts_tablet_manager.cc:1403] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.290s	user 0.010s	sys 0.000s
I20260709 13:00:56.277657  8572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ad38ee7735403281a14a504308a8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.275097  8773 raft_consensus.cc:359] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.308166  8773 raft_consensus.cc:385] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:56.309717  8717 tablet_service.cc:1511] Processing CreateTablet for tablet 2947646ebfc34bd7a5a55312f5812ee2 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:00:56.322266  8643 tablet_service.cc:1511] Processing CreateTablet for tablet bf63362bb9b4409299528319b7472b04 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:00:56.323063  8775 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:56.323989  8774 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:56.326611  8720 tablet_service.cc:1511] Processing CreateTablet for tablet bf63362bb9b4409299528319b7472b04 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:00:56.326239  8570 tablet_service.cc:1511] Processing CreateTablet for tablet a4f31f7ca2bf4416bec3531998325cc4 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:00:56.316494  8718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2142382a5d4439915e07c70837c688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.315529  8773 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:56.339144  8774 ts_tablet_manager.cc:1403] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.265s	user 0.012s	sys 0.013s
I20260709 13:00:56.338896  8775 ts_tablet_manager.cc:1403] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.294s	user 0.003s	sys 0.009s
I20260709 13:00:56.351298  8773 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.357712  8643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf63362bb9b4409299528319b7472b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.361802  8722 tablet_service.cc:1511] Processing CreateTablet for tablet bab857d69b5f4bc480340a9324923bc4 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:00:56.362980  8648 tablet_service.cc:1511] Processing CreateTablet for tablet b4572d49b22942189c377be6ba2293d6 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:00:56.369525  8648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4572d49b22942189c377be6ba2293d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.372742  8641 tablet_service.cc:1511] Processing CreateTablet for tablet a4f31f7ca2bf4416bec3531998325cc4 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:00:56.374189  8641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f31f7ca2bf4416bec3531998325cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.315389  8569 tablet_service.cc:1511] Processing CreateTablet for tablet b4572d49b22942189c377be6ba2293d6 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:00:56.381644  8568 tablet_service.cc:1511] Processing CreateTablet for tablet 7778779e8695473fa6249308e7a4fa1e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:00:56.386543  8719 tablet_service.cc:1511] Processing CreateTablet for tablet b4572d49b22942189c377be6ba2293d6 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:00:56.388476  8649 tablet_service.cc:1511] Processing CreateTablet for tablet 06f98a166dd945b39554b2f278c933ef (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:00:56.389430  8569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4572d49b22942189c377be6ba2293d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.379105  8642 tablet_service.cc:1511] Processing CreateTablet for tablet 7778779e8695473fa6249308e7a4fa1e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:00:56.372237  8774 raft_consensus.cc:359] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.385087  8571 tablet_service.cc:1511] Processing CreateTablet for tablet 06f98a166dd945b39554b2f278c933ef (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:00:56.396112  8609 heartbeater.cc:499] Master 127.7.78.126:44983 was elected leader, sending a full tablet report...
I20260709 13:00:56.397217  8646 tablet_service.cc:1511] Processing CreateTablet for tablet f4f350f118fc449fb61357528a6b14f7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:00:56.405802  8573 tablet_service.cc:1511] Processing CreateTablet for tablet f4f350f118fc449fb61357528a6b14f7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:00:56.408253  8644 tablet_service.cc:1511] Processing CreateTablet for tablet d851286f2cf248549fc81f8296784c08 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:00:56.408843  8723 tablet_service.cc:1511] Processing CreateTablet for tablet 06f98a166dd945b39554b2f278c933ef (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:00:56.393554  8775 raft_consensus.cc:359] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:56.410140  8775 raft_consensus.cc:385] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:56.410486  8775 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:56.411013  8716 tablet_service.cc:1511] Processing CreateTablet for tablet a4f31f7ca2bf4416bec3531998325cc4 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:00:56.411208  8775 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:56.412979  8715 tablet_service.cc:1511] Processing CreateTablet for tablet 7778779e8695473fa6249308e7a4fa1e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:00:56.398252  8720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf63362bb9b4409299528319b7472b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.393522  8774 raft_consensus.cc:385] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:56.420507  8649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f98a166dd945b39554b2f278c933ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.421564  8684 heartbeater.cc:499] Master 127.7.78.126:44983 was elected leader, sending a full tablet report...
I20260709 13:00:56.429646  8574 tablet_service.cc:1511] Processing CreateTablet for tablet a6c36d18684a44799bfb89a6d20e4fd7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:00:56.434574  8773 ts_tablet_manager.cc:1434] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.162s	user 0.010s	sys 0.016s
I20260709 13:00:56.435596  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f31f7ca2bf4416bec3531998325cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.436205  8722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab857d69b5f4bc480340a9324923bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.436884  8723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f98a166dd945b39554b2f278c933ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.420024  8570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f31f7ca2bf4416bec3531998325cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.437655  8717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947646ebfc34bd7a5a55312f5812ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.438082  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f98a166dd945b39554b2f278c933ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.415438  8774 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:56.439232  8774 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.439782  8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f350f118fc449fb61357528a6b14f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.441690  8568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7778779e8695473fa6249308e7a4fa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.443434  8719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4572d49b22942189c377be6ba2293d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.451521  8566 tablet_service.cc:1511] Processing CreateTablet for tablet 98c31f9999e84270898dfb87c21cbc7c (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:00:56.453038  8566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98c31f9999e84270898dfb87c21cbc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.463527  8567 tablet_service.cc:1511] Processing CreateTablet for tablet 081470ef13974fe497f88859ada43ffa (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:00:56.476728  8721 tablet_service.cc:1511] Processing CreateTablet for tablet f4f350f118fc449fb61357528a6b14f7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:00:56.478168  8721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f350f118fc449fb61357528a6b14f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.480325  8640 tablet_service.cc:1511] Processing CreateTablet for tablet a6c36d18684a44799bfb89a6d20e4fd7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:00:56.481463  8642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7778779e8695473fa6249308e7a4fa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.482127  8640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6c36d18684a44799bfb89a6d20e4fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.486966  8715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7778779e8695473fa6249308e7a4fa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.487327  8572 tablet_service.cc:1511] Processing CreateTablet for tablet 96614bb40dc54bb2bdcff8f9d5476bb1 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:00:56.489935  8644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d851286f2cf248549fc81f8296784c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.491178  8645 tablet_service.cc:1511] Processing CreateTablet for tablet 96614bb40dc54bb2bdcff8f9d5476bb1 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:00:56.492528  8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96614bb40dc54bb2bdcff8f9d5476bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.493655  8565 tablet_service.cc:1511] Processing CreateTablet for tablet d851286f2cf248549fc81f8296784c08 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:00:56.494520  8773 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:56.495386  8647 tablet_service.cc:1511] Processing CreateTablet for tablet 98c31f9999e84270898dfb87c21cbc7c (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:00:56.496582  8647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98c31f9999e84270898dfb87c21cbc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.500000  8758 heartbeater.cc:499] Master 127.7.78.126:44983 was elected leader, sending a full tablet report...
I20260709 13:00:56.503631  8780 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.504074  8648 tablet_service.cc:1511] Processing CreateTablet for tablet 081470ef13974fe497f88859ada43ffa (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:00:56.504766  8774 ts_tablet_manager.cc:1434] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.156s	user 0.008s	sys 0.004s
I20260709 13:00:56.506343  8775 ts_tablet_manager.cc:1434] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.158s	user 0.008s	sys 0.003s
I20260709 13:00:56.507396  8775 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:56.504114  8780 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:56.512897  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f350f118fc449fb61357528a6b14f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.515256  8648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081470ef13974fe497f88859ada43ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.518002  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6c36d18684a44799bfb89a6d20e4fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.525142  8774 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:56.528548  8572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96614bb40dc54bb2bdcff8f9d5476bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.530092  8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d851286f2cf248549fc81f8296784c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.533604  8567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081470ef13974fe497f88859ada43ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.552172  8773 tablet_bootstrap.cc:654] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:56.574431  8775 tablet_bootstrap.cc:654] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:56.582276  8571 tablet_service.cc:1511] Processing CreateTablet for tablet 94c0917e28b9498a823f74b5f664a815 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:00:56.583791  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94c0917e28b9498a823f74b5f664a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.591965  8781 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.592444  8781 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.592607  8723 tablet_service.cc:1511] Processing CreateTablet for tablet a6c36d18684a44799bfb89a6d20e4fd7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:00:56.593928  8723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6c36d18684a44799bfb89a6d20e4fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.594394  8573 tablet_service.cc:1511] Processing CreateTablet for tablet 77d6200607074a5ebd3e4ae470a2d6ed (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:00:56.594951  8781 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:56.595589  8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d6200607074a5ebd3e4ae470a2d6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.572906  8714 tablet_service.cc:1511] Processing CreateTablet for tablet d851286f2cf248549fc81f8296784c08 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:00:56.600176  8780 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:56.586354  8648 tablet_service.cc:1511] Processing CreateTablet for tablet 94c0917e28b9498a823f74b5f664a815 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:00:56.637029  8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d851286f2cf248549fc81f8296784c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.648700  8717 tablet_service.cc:1511] Processing CreateTablet for tablet 94c0917e28b9498a823f74b5f664a815 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:00:56.661425  8565 tablet_service.cc:1511] Processing CreateTablet for tablet 01467ec5f29c4a728b24e55d1f5d2e94 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:00:56.662640  8717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94c0917e28b9498a823f74b5f664a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.649986  8718 tablet_service.cc:1511] Processing CreateTablet for tablet 081470ef13974fe497f88859ada43ffa (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:00:56.644120  8648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94c0917e28b9498a823f74b5f664a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.660745  8721 tablet_service.cc:1511] Processing CreateTablet for tablet 96614bb40dc54bb2bdcff8f9d5476bb1 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:00:56.682050  8721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96614bb40dc54bb2bdcff8f9d5476bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.631928  8719 tablet_service.cc:1511] Processing CreateTablet for tablet 98c31f9999e84270898dfb87c21cbc7c (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:00:56.684028  8641 tablet_service.cc:1511] Processing CreateTablet for tablet cde36bf9fcc5409e8a00b747fbfbfb06 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:00:56.695740  8643 tablet_service.cc:1511] Processing CreateTablet for tablet c3d38a98ca324ddbbb3ff25fb6ded965 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:00:56.696782  8641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde36bf9fcc5409e8a00b747fbfbfb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.687932  8774 tablet_bootstrap.cc:654] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:56.698360  8722 tablet_service.cc:1511] Processing CreateTablet for tablet 77d6200607074a5ebd3e4ae470a2d6ed (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:00:56.691540  8570 tablet_service.cc:1511] Processing CreateTablet for tablet cde36bf9fcc5409e8a00b747fbfbfb06 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:00:56.689352  8719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98c31f9999e84270898dfb87c21cbc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.705894  8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01467ec5f29c4a728b24e55d1f5d2e94. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:00:56.717979  8468 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44875.4us
I20260709 13:00:56.720625  8716 tablet_service.cc:1511] Processing CreateTablet for tablet cde36bf9fcc5409e8a00b747fbfbfb06 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:00:56.722327  8720 tablet_service.cc:1511] Processing CreateTablet for tablet 01467ec5f29c4a728b24e55d1f5d2e94 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:00:56.722983  8773 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:56.728320  8570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde36bf9fcc5409e8a00b747fbfbfb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.686537  8640 tablet_service.cc:1511] Processing CreateTablet for tablet b9b3791a4a534a598bc36d62992d27be (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:00:56.725351  8773 ts_tablet_manager.cc:1403] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.231s	user 0.015s	sys 0.029s
I20260709 13:00:56.734638  8649 tablet_service.cc:1511] Processing CreateTablet for tablet 77d6200607074a5ebd3e4ae470a2d6ed (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:00:56.737872  8773 raft_consensus.cc:359] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.738617  8773 raft_consensus.cc:385] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:56.738519  8642 tablet_service.cc:1511] Processing CreateTablet for tablet 01467ec5f29c4a728b24e55d1f5d2e94 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:00:56.739001  8773 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:56.742859  8569 tablet_service.cc:1511] Processing CreateTablet for tablet c3d38a98ca324ddbbb3ff25fb6ded965 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:00:56.744258  8569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d38a98ca324ddbbb3ff25fb6ded965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.724795  8643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d38a98ca324ddbbb3ff25fb6ded965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.750065  8718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081470ef13974fe497f88859ada43ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.755144  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:56.755214  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:56.760119  8549 leader_election.cc:343] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:56.761059  8550 leader_election.cc:343] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:56.761528  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:56.762650  8781 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:56.759846  8715 tablet_service.cc:1511] Processing CreateTablet for tablet b9b3791a4a534a598bc36d62992d27be (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:00:56.755350  8574 tablet_service.cc:1511] Processing CreateTablet for tablet dd0fc940d95d4b118d3e2564ed48b09e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:00:56.765190  8773 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.774889  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde36bf9fcc5409e8a00b747fbfbfb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.781786  8642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01467ec5f29c4a728b24e55d1f5d2e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.787951  8722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d6200607074a5ebd3e4ae470a2d6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.790405  8568 tablet_service.cc:1511] Processing CreateTablet for tablet 9f9dd5b88e0647f284658784381e1893 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:00:56.791258  8775 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:56.791879  8720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01467ec5f29c4a728b24e55d1f5d2e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.792110  8640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b3791a4a534a598bc36d62992d27be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.784965  8566 tablet_service.cc:1511] Processing CreateTablet for tablet b9b3791a4a534a598bc36d62992d27be (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:00:56.796926  8715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b3791a4a534a598bc36d62992d27be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.791870  8775 ts_tablet_manager.cc:1403] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.285s	user 0.014s	sys 0.011s
I20260709 13:00:56.803393  8646 tablet_service.cc:1511] Processing CreateTablet for tablet dd0fc940d95d4b118d3e2564ed48b09e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:00:56.803915  8649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d6200607074a5ebd3e4ae470a2d6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.811832  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd0fc940d95d4b118d3e2564ed48b09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.808964  8775 raft_consensus.cc:359] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.814945  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd0fc940d95d4b118d3e2564ed48b09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.815629  8773 ts_tablet_manager.cc:1434] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.081s	user 0.007s	sys 0.001s
I20260709 13:00:56.818257  8775 raft_consensus.cc:385] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:56.829787  8779 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.832789  8779 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.833477  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:00:56.835019  8696 leader_election.cc:343] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:56.835916  8567 tablet_service.cc:1511] Processing CreateTablet for tablet 04399390cf414886a3453981dc1f74aa (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:00:56.838511  8647 tablet_service.cc:1511] Processing CreateTablet for tablet 04399390cf414886a3453981dc1f74aa (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:00:56.840238  8647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04399390cf414886a3453981dc1f74aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.840749  8567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04399390cf414886a3453981dc1f74aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.841812  8566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b3791a4a534a598bc36d62992d27be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.844813  8779 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:56.867523  8568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9dd5b88e0647f284658784381e1893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.832759  8645 tablet_service.cc:1511] Processing CreateTablet for tablet 9f9dd5b88e0647f284658784381e1893 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:00:56.873579  8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9dd5b88e0647f284658784381e1893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.879787  8574 tablet_service.cc:1511] Processing CreateTablet for tablet 294596405ca140769bf30fa7ee1e4820 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:00:56.881999  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294596405ca140769bf30fa7ee1e4820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.882148  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:56.884598  8698 leader_election.cc:343] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:56.885111  8698 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:56.886307  8780 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:56.886715  8644 tablet_service.cc:1511] Processing CreateTablet for tablet 9a389d19cf2c47d6b5873139d8cfad40 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:00:56.888060  8644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a389d19cf2c47d6b5873139d8cfad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.877437  8721 tablet_service.cc:1511] Processing CreateTablet for tablet c3d38a98ca324ddbbb3ff25fb6ded965 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:00:56.890058  8721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d38a98ca324ddbbb3ff25fb6ded965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.831667  8775 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:56.897464  8775 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.899685  8775 ts_tablet_manager.cc:1434] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.096s	user 0.002s	sys 0.004s
I20260709 13:00:56.900743  8775 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:56.901727  8773 tablet_bootstrap.cc:492] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:56.906391  8572 tablet_service.cc:1511] Processing CreateTablet for tablet 9a389d19cf2c47d6b5873139d8cfad40 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:00:56.907740  8572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a389d19cf2c47d6b5873139d8cfad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.921782  8779 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.922446  8779 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.945250  8780 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:56.945844  8780 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:56.921180  8571 tablet_service.cc:1511] Processing CreateTablet for tablet 652ad3041b404044aaf40f77c1be3a32 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:00:56.971634  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652ad3041b404044aaf40f77c1be3a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:56.996310  8779 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:00:56.997020  8466 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 76698.9us
I20260709 13:00:57.002826  8774 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:57.003450  8774 ts_tablet_manager.cc:1403] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.479s	user 0.012s	sys 0.034s
I20260709 13:00:57.019413  8574 tablet_service.cc:1511] Processing CreateTablet for tablet 9e8a9411169f407cba1b44443edc9840 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:00:57.021474  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8a9411169f407cba1b44443edc9840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.032487  8774 raft_consensus.cc:359] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.033275  8774 raft_consensus.cc:385] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.033598  8774 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.034260  8774 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.037446  8774 ts_tablet_manager.cc:1434] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.034s	user 0.017s	sys 0.008s
I20260709 13:00:57.038477  8774 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:57.041357  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:57.045027  8774 tablet_bootstrap.cc:654] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.047860  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:57.048605  8659 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:57.050658  8698 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:57.062017  8780 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:57.070641  8780 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:57.071170  8780 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.071678  8780 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.074379  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:00:57.076624  8622 leader_election.cc:343] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.063699  8773 tablet_bootstrap.cc:654] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:57.064648  8696 leader_election.cc:343] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.066073  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:57.083683  8584 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:00:57.066185  8574 tablet_service.cc:1511] Processing CreateTablet for tablet 7eaeef2720084b6aac897de8d33f6921 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:00:57.093421  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eaeef2720084b6aac897de8d33f6921. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:00:57.107883  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57317.4us
I20260709 13:00:57.112627  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:57.114051  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:00:57.115489  8625 leader_election.cc:343] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.116727  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:00:57.118081  8779 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:57.118578  8779 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.119009  8779 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.135912  8774 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:57.135850  8780 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.136505  8774 ts_tablet_manager.cc:1403] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.098s	user 0.056s	sys 0.020s
I20260709 13:00:57.136801  8714 tablet_service.cc:1511] Processing CreateTablet for tablet 9f9dd5b88e0647f284658784381e1893 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:00:57.137472  8732 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:00:57.138252  8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9dd5b88e0647f284658784381e1893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.139906  8774 raft_consensus.cc:359] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.140638  8774 raft_consensus.cc:385] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.140951  8774 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.140856  8625 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:00:57.155081  8775 tablet_bootstrap.cc:654] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.154886  8641 tablet_service.cc:1511] Processing CreateTablet for tablet 294596405ca140769bf30fa7ee1e4820 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:00:57.156216  8641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294596405ca140769bf30fa7ee1e4820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.156104  8779 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.158638  8780 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:57.159875  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:57.159703  8573 tablet_service.cc:1511] Processing CreateTablet for tablet 9413ba3105764883ae0d7597a86ad605 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:00:57.161303  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:57.161899  8659 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.161679  8565 tablet_service.cc:1511] Processing CreateTablet for tablet 96ad7a91a5c14870885cd58a34393f82 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:00:57.161988  8573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9413ba3105764883ae0d7597a86ad605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.164093  8774 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.177646  8584 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:57.185770  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:57.169919  8720 tablet_service.cc:1511] Processing CreateTablet for tablet dd0fc940d95d4b118d3e2564ed48b09e (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:00:57.172689  8779 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:57.187453  8720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd0fc940d95d4b118d3e2564ed48b09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.188627  8584 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:57.195446  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:57.196058  8583 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.201215  8570 tablet_service.cc:1511] Processing CreateTablet for tablet d35f944458a44f6c82498c7fec026ec5 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:00:57.204429  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:57.215888  8780 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:57.216404  8780 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:57.217195  8780 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.206204  8649 tablet_service.cc:1511] Processing CreateTablet for tablet 9e8a9411169f407cba1b44443edc9840 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:00:57.173885  8645 tablet_service.cc:1511] Processing CreateTablet for tablet 652ad3041b404044aaf40f77c1be3a32 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:00:57.210187  8568 tablet_service.cc:1511] Processing CreateTablet for tablet 3ad2aa6046e746348b02a2571b7580f0 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:00:57.212571  8583 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:00:57.213929  8565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ad7a91a5c14870885cd58a34393f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.214437  8721 tablet_service.cc:1511] Processing CreateTablet for tablet 04399390cf414886a3453981dc1f74aa (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:00:57.209093  8774 ts_tablet_manager.cc:1434] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.072s	user 0.008s	sys 0.000s
I20260709 13:00:57.236184  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:00:57.230238  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 89502.8us
I20260709 13:00:57.245301  8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652ad3041b404044aaf40f77c1be3a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.248833  8797 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:57.248997  8721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04399390cf414886a3453981dc1f74aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.246771  8568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ad2aa6046e746348b02a2571b7580f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.259840  8642 tablet_service.cc:1511] Processing CreateTablet for tablet 7eaeef2720084b6aac897de8d33f6921 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:00:57.261468  8717 tablet_service.cc:1511] Processing CreateTablet for tablet 9a389d19cf2c47d6b5873139d8cfad40 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:00:57.262769  8717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a389d19cf2c47d6b5873139d8cfad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.265591  8774 tablet_bootstrap.cc:492] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:57.239494  8571 tablet_service.cc:1511] Processing CreateTablet for tablet 96cbbc16390646a9875b38f1b174f981 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:00:57.253202  8659 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:57.276315  8797 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:00:57.287955  8797 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.281044  8779 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:57.292582  8646 tablet_service.cc:1511] Processing CreateTablet for tablet 96ad7a91a5c14870885cd58a34393f82 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:00:57.296283  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:57.289567  8649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8a9411169f407cba1b44443edc9840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.301065  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ad7a91a5c14870885cd58a34393f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.302127  8775 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:57.302615  8775 ts_tablet_manager.cc:1403] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.402s	user 0.017s	sys 0.004s
I20260709 13:00:57.304529  8480 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:57.306438  8569 tablet_service.cc:1511] Processing CreateTablet for tablet b8b894d0fc374f33a626d5fd9d8ca4e7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:00:57.308228  8570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35f944458a44f6c82498c7fec026ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.309345  8647 tablet_service.cc:1511] Processing CreateTablet for tablet d35f944458a44f6c82498c7fec026ec5 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:00:57.310751  8647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35f944458a44f6c82498c7fec026ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.311373  8567 tablet_service.cc:1511] Processing CreateTablet for tablet 2adcf8a687b54060b644e8c691d8772f (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:00:57.312582  8567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2adcf8a687b54060b644e8c691d8772f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.313961  8648 tablet_service.cc:1511] Processing CreateTablet for tablet 3ad2aa6046e746348b02a2571b7580f0 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:00:57.315210  8648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ad2aa6046e746348b02a2571b7580f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.319074  8774 tablet_bootstrap.cc:654] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.318938  8566 tablet_service.cc:1511] Processing CreateTablet for tablet 4d18196b7e714cb6a79ec64ea3d65803 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:00:57.319461  8642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eaeef2720084b6aac897de8d33f6921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.330619  8714 tablet_service.cc:1511] Processing CreateTablet for tablet 652ad3041b404044aaf40f77c1be3a32 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:00:57.342012  8721 tablet_service.cc:1511] Processing CreateTablet for tablet 9e8a9411169f407cba1b44443edc9840 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:00:57.342907  8715 tablet_service.cc:1511] Processing CreateTablet for tablet 294596405ca140769bf30fa7ee1e4820 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:00:57.344014  8723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 32) took 1109 ms. Trace:
I20260709 13:00:57.344467  8723 rpcz_store.cc:276] 0709 13:00:56.234908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.591603 (+356695us) service_pool.cc:224] Handling call
0709 13:00:56.593463 (+  1860us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:56.593560 (+    97us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:56.821075 (+227515us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.336434 (+515359us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.343953 (+  7519us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.346580  8574 tablet_service.cc:1511] Processing CreateTablet for tablet 90d5e31850d64b9eb8c754954c889846 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:00:57.342012  8643 tablet_service.cc:1511] Processing CreateTablet for tablet 9413ba3105764883ae0d7597a86ad605 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:00:57.348970  8643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9413ba3105764883ae0d7597a86ad605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.353624  8775 raft_consensus.cc:359] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.354374  8775 raft_consensus.cc:385] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.354691  8775 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.347059  8723 tablet_service.cc:1511] Processing CreateTablet for tablet 7eaeef2720084b6aac897de8d33f6921 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:00:57.316879  8718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 35) took 1059 ms. Trace:
I20260709 13:00:57.365552  8773 tablet_bootstrap.cc:492] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:57.383260  8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652ad3041b404044aaf40f77c1be3a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.362514  8571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96cbbc16390646a9875b38f1b174f981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.386308  8716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 39) took 1104 ms. Trace:
I20260709 13:00:57.386868  8716 rpcz_store.cc:276] 0709 13:00:56.282257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.719620 (+437363us) service_pool.cc:224] Handling call
0709 13:00:56.729544 (+  9924us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:56.729650 (+   106us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.179014 (+449364us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.332502 (+153488us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.386213 (+ 53711us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.382014  8773 ts_tablet_manager.cc:1403] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.481s	user 0.020s	sys 0.008s
I20260709 13:00:57.390902  8482 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:57.393891  8716 tablet_service.cc:1511] Processing CreateTablet for tablet 3ad2aa6046e746348b02a2571b7580f0 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:00:57.396456  8773 raft_consensus.cc:359] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.362253  8718 rpcz_store.cc:276] 0709 13:00:56.257187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.615104 (+357917us) service_pool.cc:224] Handling call
0709 13:00:56.701184 (+ 86080us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:56.707762 (+  6578us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.154338 (+446576us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.277029 (+122691us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.316365 (+ 39336us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.362076  8644 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 45) took 1008 ms. Trace:
I20260709 13:00:57.401750  8644 rpcz_store.cc:276] 0709 13:00:56.354039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.885690 (+531651us) service_pool.cc:224] Handling call
0709 13:00:56.887573 (+  1883us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:56.887672 (+    99us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.175701 (+288029us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.325237 (+149536us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.362008 (+ 36771us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.404049  8644 tablet_service.cc:1511] Processing CreateTablet for tablet 96cbbc16390646a9875b38f1b174f981 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:00:57.405504  8644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96cbbc16390646a9875b38f1b174f981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.406850  8641 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 46) took 1040 ms. Trace:
I20260709 13:00:57.407312  8641 rpcz_store.cc:276] 0709 13:00:56.366525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.153904 (+787379us) service_pool.cc:224] Handling call
0709 13:00:57.155758 (+  1854us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.155855 (+    97us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.398607 (+242752us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.406223 (+  7616us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.406780 (+   557us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.410305  8641 tablet_service.cc:1511] Processing CreateTablet for tablet 4d18196b7e714cb6a79ec64ea3d65803 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:00:57.411737  8641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d18196b7e714cb6a79ec64ea3d65803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.414594  8647 tablet_service.cc:1511] Processing CreateTablet for tablet 90d5e31850d64b9eb8c754954c889846 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:00:57.416007  8647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d5e31850d64b9eb8c754954c889846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.418048  8566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d18196b7e714cb6a79ec64ea3d65803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.397213  8773 raft_consensus.cc:385] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:00:57.378532  8625 leader_election.cc:343] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.423013  8574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d5e31850d64b9eb8c754954c889846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.427557  8569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b894d0fc374f33a626d5fd9d8ca4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.427953  8715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294596405ca140769bf30fa7ee1e4820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.424050  8773 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.431823  8773 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.434469  8773 ts_tablet_manager.cc:1434] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.043s	user 0.008s	sys 0.000s
I20260709 13:00:57.435649  8773 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:57.438971  8720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 42) took 1119 ms. Trace:
I20260709 13:00:57.477118  8720 rpcz_store.cc:276] 0709 13:00:56.319935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.108981 (+789046us) service_pool.cc:224] Handling call
0709 13:00:57.186792 (+ 77811us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.186902 (+   110us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.353364 (+166462us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.438374 (+ 85010us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.438904 (+   530us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.478436  8646 tablet_service.cc:1511] Processing CreateTablet for tablet b8b894d0fc374f33a626d5fd9d8ca4e7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:00:57.479344  8720 tablet_service.cc:1511] Processing CreateTablet for tablet 96ad7a91a5c14870885cd58a34393f82 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:00:57.447062  8723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eaeef2720084b6aac897de8d33f6921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.480435  8721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8a9411169f407cba1b44443edc9840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.481474  8645 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 47) took 1098 ms. Trace:
I20260709 13:00:57.481894  8645 rpcz_store.cc:276] 0709 13:00:56.383375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.172630 (+789255us) service_pool.cc:224] Handling call
0709 13:00:57.226823 (+ 54193us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.229119 (+  2296us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.398027 (+168908us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.480990 (+ 82963us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.481353 (+   363us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.442461  8714 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 47) took 1057 ms. Trace:
W20260709 13:00:57.453657  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 94129.8us
I20260709 13:00:57.401062  8718 tablet_service.cc:1511] Processing CreateTablet for tablet 9413ba3105764883ae0d7597a86ad605 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:00:57.485740  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ad2aa6046e746348b02a2571b7580f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.486647  8718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9413ba3105764883ae0d7597a86ad605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.486894  8649 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 48) took 1065 ms. Trace:
I20260709 13:00:57.487874  8773 tablet_bootstrap.cc:654] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.487322  8649 rpcz_store.cc:276] 0709 13:00:56.421104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.205173 (+784069us) service_pool.cc:224] Handling call
0709 13:00:57.279351 (+ 74178us) spinlock_profiling.cc:243] Waited 997 us on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec4a831 00007f140ec27ac3 00007f140ebd6400 00007f140805baf4 00007f140805ba82 00007f140805b9b1
0709 13:00:57.282434 (+  3083us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.282659 (+   225us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.417898 (+135239us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.471874 (+ 53976us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.486822 (+ 14948us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2292480}
I20260709 13:00:57.463223  8774 tablet_bootstrap.cc:492] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:57.467430  8640 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 40) took 1181 ms. Trace:
I20260709 13:00:57.489753  8774 ts_tablet_manager.cc:1403] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.235s	user 0.015s	sys 0.006s
I20260709 13:00:57.489637  8640 rpcz_store.cc:276] 0709 13:00:56.285897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.685579 (+399682us) service_pool.cc:224] Handling call
0709 13:00:56.734485 (+ 48906us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:56.734583 (+    98us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.220844 (+486261us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.466840 (+245996us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.467356 (+   516us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.490500  8717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 45) took 1127 ms. Trace:
I20260709 13:00:57.490969  8717 rpcz_store.cc:276] 0709 13:00:56.362681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.260537 (+897856us) service_pool.cc:224] Handling call
0709 13:00:57.262267 (+  1730us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.262377 (+   110us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.381104 (+118727us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.460588 (+ 79484us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.490337 (+ 29749us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.378459  8775 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.495611  8775 ts_tablet_manager.cc:1434] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.193s	user 0.006s	sys 0.000s
I20260709 13:00:57.493026  8780 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.474766  8719 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 34) took 1223 ms. Trace:
I20260709 13:00:57.497417  8780 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.497502  8719 rpcz_store.cc:276] 0709 13:00:56.251225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.611968 (+360743us) service_pool.cc:224] Handling call
0709 13:00:56.688861 (+ 76893us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:56.688967 (+   106us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.159541 (+470574us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.473782 (+314241us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.474623 (+   841us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.500713  8719 tablet_service.cc:1511] Processing CreateTablet for tablet d35f944458a44f6c82498c7fec026ec5 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:00:57.479725  8646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b894d0fc374f33a626d5fd9d8ca4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.424504  8722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 37) took 1155 ms. Trace:
I20260709 13:00:57.502322  8719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35f944458a44f6c82498c7fec026ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.501873  8722 rpcz_store.cc:276] 0709 13:00:56.269289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:56.657359 (+388070us) service_pool.cc:224] Handling call
0709 13:00:56.699230 (+ 41871us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:56.699327 (+    97us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.196950 (+497623us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.354534 (+157584us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.423930 (+ 69396us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.505534  8722 tablet_service.cc:1511] Processing CreateTablet for tablet 96cbbc16390646a9875b38f1b174f981 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:00:57.484274  8714 rpcz_store.cc:276] 0709 13:00:56.385474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.329592 (+944118us) service_pool.cc:224] Handling call
0709 13:00:57.378925 (+ 49333us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.379051 (+   126us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.393900 (+ 14849us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.400751 (+  6851us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.442034 (+ 41283us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af21a2 00007f140fa33bb9 00007f140eb64d88 00007f140ec2b973 00007f140ec2770f 00007f140ec27ecd 00007f140ebd6400 00007f140805baf4 00007f140805ba82
0709 13:00:57.442394 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49261952}
I20260709 13:00:57.475419  8642 tablet_service.cc:1511] Processing CreateTablet for tablet 2adcf8a687b54060b644e8c691d8772f (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:00:57.508705  8714 tablet_service.cc:1511] Processing CreateTablet for tablet 4d18196b7e714cb6a79ec64ea3d65803 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:00:57.520587  8717 tablet_service.cc:1511] Processing CreateTablet for tablet 90d5e31850d64b9eb8c754954c889846 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:00:57.524746  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:57.525583  8780 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:57.526199  8642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2adcf8a687b54060b644e8c691d8772f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:00:57.526330  8696 leader_election.cc:343] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.530682  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:57.532308  8698 leader_election.cc:343] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.532779  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:57.533782  8775 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:57.534642  8722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96cbbc16390646a9875b38f1b174f981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.535364  8720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ad7a91a5c14870885cd58a34393f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.540738  8714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d18196b7e714cb6a79ec64ea3d65803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.543704  8717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d5e31850d64b9eb8c754954c889846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.550156  8774 raft_consensus.cc:359] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.551226  8774 raft_consensus.cc:385] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.551623  8774 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.552618  8774 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.554143  8780 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:57.572788  8648 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 50) took 1012 ms. Trace:
I20260709 13:00:57.573355  8648 rpcz_store.cc:276] 0709 13:00:56.560172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.268057 (+707885us) service_pool.cc:224] Handling call
0709 13:00:57.314761 (+ 46704us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.314854 (+    93us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.448458 (+133604us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.572053 (+123595us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.572712 (+   659us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.575809  8774 ts_tablet_manager.cc:1434] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.086s	user 0.007s	sys 0.005s
I20260709 13:00:57.581476  8775 tablet_bootstrap.cc:654] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.586694  8715 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 46) took 1214 ms. Trace:
I20260709 13:00:57.593295  8715 rpcz_store.cc:276] 0709 13:00:56.372634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.297952 (+925318us) service_pool.cc:224] Handling call
0709 13:00:57.421724 (+123772us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.422943 (+  1219us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.463821 (+ 40878us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.583166 (+119345us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.586622 (+  3456us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.605542  8774 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:57.609331  8780 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:00:57.624806  8723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 49) took 1111 ms. Trace:
I20260709 13:00:57.634445  8774 tablet_bootstrap.cc:654] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.635165  8775 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:57.637676  8718 tablet_service.cc:1511] Processing CreateTablet for tablet b8b894d0fc374f33a626d5fd9d8ca4e7 (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:00:57.646018  8775 ts_tablet_manager.cc:1403] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.149s	user 0.011s	sys 0.013s
I20260709 13:00:57.646869  8718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b894d0fc374f33a626d5fd9d8ca4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.639197  8716 tablet_service.cc:1511] Processing CreateTablet for tablet 2adcf8a687b54060b644e8c691d8772f (DEFAULT_TABLE table=test-workload [id=a52f92b6e8d04877ae60579197efba09]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:00:57.640244  8780 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:57.651002  8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2adcf8a687b54060b644e8c691d8772f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:57.650890  8775 raft_consensus.cc:359] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.652366  8775 raft_consensus.cc:385] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.652825  8775 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.665684  8773 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:57.666303  8773 ts_tablet_manager.cc:1403] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.231s	user 0.036s	sys 0.023s
I20260709 13:00:57.667388  8721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40398 (request call id 48) took 1180 ms. Trace:
I20260709 13:00:57.667827  8721 rpcz_store.cc:276] 0709 13:00:56.486556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.341089 (+854533us) service_pool.cc:224] Handling call
0709 13:00:57.460740 (+119651us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af21a2 00007f140fa33bb9 00007f140eb64d88 00007f140ec27b80 00007f140ebd6400 00007f140805baf4 00007f140805ba82 00007f140805b9b1 00007f140805b92c
0709 13:00:57.460809 (+    69us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.460944 (+   135us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.568687 (+107743us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.666819 (+ 98132us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.667318 (+   499us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45998464}
I20260709 13:00:57.654302  8723 rpcz_store.cc:276] 0709 13:00:56.513712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:00:57.346060 (+832348us) service_pool.cc:224] Handling call
0709 13:00:57.423180 (+ 77120us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 13:00:57.425825 (+  2645us) ts_tablet_manager.cc:682] Creating new metadata...
0709 13:00:57.576615 (+150790us) tablet_metadata.cc:665] Metadata flushed
0709 13:00:57.624031 (+ 47416us) tablet_replica.cc:206] Creating consensus instance
0709 13:00:57.624714 (+   683us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:00:57.661970  8775 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.669467  8773 raft_consensus.cc:359] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.689345  8773 raft_consensus.cc:385] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.690070  8773 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.698179  8774 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:57.696188  8773 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.706024  8774 ts_tablet_manager.cc:1403] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.101s	user 0.014s	sys 0.006s
I20260709 13:00:57.714841  8774 raft_consensus.cc:359] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.716035  8774 raft_consensus.cc:385] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.716511  8774 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.717491  8774 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.720577  8774 ts_tablet_manager.cc:1434] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:00:57.727586  8774 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:57.735883  8775 ts_tablet_manager.cc:1434] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.089s	user 0.014s	sys 0.001s
I20260709 13:00:57.737061  8775 tablet_bootstrap.cc:492] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:57.737726  8773 ts_tablet_manager.cc:1434] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.071s	user 0.008s	sys 0.001s
I20260709 13:00:57.738878  8773 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:57.748418  8773 tablet_bootstrap.cc:654] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.737182  8774 tablet_bootstrap.cc:654] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.751041  8775 tablet_bootstrap.cc:654] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.755894  8817 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.756647  8817 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.760015  8775 tablet_bootstrap.cc:492] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:57.760574  8775 ts_tablet_manager.cc:1403] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.005s
I20260709 13:00:57.761971  8732 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.763500  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:57.764597  8583 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 1: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:00:57.766264  8817 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:57.763703  8775 raft_consensus.cc:359] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.767119  8774 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:57.766990  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:57.767366  8775 raft_consensus.cc:385] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.767689  8774 ts_tablet_manager.cc:1403] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.040s	user 0.007s	sys 0.019s
I20260709 13:00:57.767765  8775 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.768780  8698 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:57.768739  8775 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.770419  8796 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:57.771203  8775 ts_tablet_manager.cc:1434] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.004s
I20260709 13:00:57.770835  8774 raft_consensus.cc:359] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:57.771661  8774 raft_consensus.cc:385] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.771982  8774 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.772184  8775 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:57.772814  8774 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:57.775044  8774 ts_tablet_manager.cc:1434] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 13:00:57.777281  8774 tablet_bootstrap.cc:492] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:57.778100  8775 tablet_bootstrap.cc:654] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.783607  8797 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:57.784044  8774 tablet_bootstrap.cc:654] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.797569  8773 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:57.798295  8773 ts_tablet_manager.cc:1403] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.060s	user 0.011s	sys 0.007s
I20260709 13:00:57.802682  8773 raft_consensus.cc:359] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.803771  8773 raft_consensus.cc:385] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.804165  8773 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.805434  8773 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.817795  8774 tablet_bootstrap.cc:492] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:57.818320  8774 ts_tablet_manager.cc:1403] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.041s	user 0.019s	sys 0.018s
I20260709 13:00:57.819401  8775 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:57.819837  8775 ts_tablet_manager.cc:1403] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.048s	user 0.023s	sys 0.016s
I20260709 13:00:57.821200  8774 raft_consensus.cc:359] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.822067  8774 raft_consensus.cc:385] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.822371  8774 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.822724  8775 raft_consensus.cc:359] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.823422  8774 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.823992  8775 raft_consensus.cc:385] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.825425  8773 ts_tablet_manager.cc:1434] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.000s
I20260709 13:00:57.825564  8775 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.826807  8826 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.826498  8775 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.827270  8826 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.829008  8775 ts_tablet_manager.cc:1434] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 13:00:57.829470  8826 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:57.830132  8775 tablet_bootstrap.cc:492] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:57.831104  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:00:57.833169  8622 leader_election.cc:343] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.835716  8826 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:57.837136  8773 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:57.847280  8774 ts_tablet_manager.cc:1434] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 13:00:57.847591  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:57.848436  8774 tablet_bootstrap.cc:492] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:57.850881  8773 tablet_bootstrap.cc:654] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:57.854753  8625 leader_election.cc:343] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.855293  8625 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:00:57.855973  8774 tablet_bootstrap.cc:654] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.856218  8828 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:57.859144  8775 tablet_bootstrap.cc:654] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.865067  8781 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.865792  8781 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:57.869511  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:57.871340  8732 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:57.873077  8550 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:57.875893  8781 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:57.878964  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:57.879966  8781 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:57.880641  8549 leader_election.cc:343] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.881376  8781 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.886417  8781 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:57.887337  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:57.891883  8781 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:00:57.892652  8549 leader_election.cc:343] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.897531  8781 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.901865  8781 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.914700  8834 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.915630  8834 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.918697  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:00:57.920996  8550 leader_election.cc:343] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.921689  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:57.923384  8833 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:57.934788  8834 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:57.934994  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:57.941907  8658 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:57.943816  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:57.936251  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:57.946362  8833 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:57.946940  8833 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.947420  8833 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:57.950839  8550 leader_election.cc:343] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.941000  8781 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:57.960466  8773 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:57.961133  8775 tablet_bootstrap.cc:492] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:57.961252  8773 ts_tablet_manager.cc:1403] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.124s	user 0.054s	sys 0.060s
I20260709 13:00:57.961869  8775 ts_tablet_manager.cc:1403] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.132s	user 0.039s	sys 0.052s
I20260709 13:00:57.962396  8774 tablet_bootstrap.cc:492] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:57.962847  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:57.963490  8732 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.964581  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:57.964423  8773 raft_consensus.cc:359] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.965212  8773 raft_consensus.cc:385] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.964967  8775 raft_consensus.cc:359] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.965581  8773 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.966045  8775 raft_consensus.cc:385] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:00:57.966149  8549 leader_election.cc:343] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.966410  8775 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.966770  8781 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:57.966645  8773 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.967492  8817 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.967387  8775 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.968031  8817 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.969889  8775 ts_tablet_manager.cc:1434] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 13:00:57.969343  8833 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.970216  8817 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:57.962847  8774 ts_tablet_manager.cc:1403] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.115s	user 0.030s	sys 0.081s
I20260709 13:00:57.971719  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:57.972208  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:57.972353  8584 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:57.972841  8658 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:57.973686  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:57.974476  8817 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:57.974247  8774 raft_consensus.cc:359] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.974828  8817 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:57.974922  8774 raft_consensus.cc:385] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:57.975230  8817 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.975337  8774 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:57.976227  8774 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.978075  8775 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:57.979738  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:57.980360  8658 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.980763  8833 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:57.981720  8833 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:57.982288  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:57.984856  8817 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:57.986083  8550 leader_election.cc:343] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.983027  8774 ts_tablet_manager.cc:1434] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260709 13:00:57.988155  8774 tablet_bootstrap.cc:492] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:57.989625  8775 tablet_bootstrap.cc:654] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:57.990214  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:57.984417  8732 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:00:57.987973  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:57.990851  8659 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.990991  8584 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:57.991602  8550 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:57.992420  8781 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:57.986763  8833 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:57.992861  8781 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:00:57.993714  8781 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:57.995545  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:57.995671  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:57.996912  8550 leader_election.cc:343] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:57.997735  8833 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:57.999636  8658 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
W20260709 13:00:57.999936  8549 leader_election.cc:343] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.000401  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.001806  8774 tablet_bootstrap.cc:654] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.002043  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:00:57.987061  8817 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.003278  8834 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.003985  8834 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.004403  8834 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:00:58.002853  8773 ts_tablet_manager.cc:1434] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.041s	user 0.006s	sys 0.000s
I20260709 13:00:58.005252  8834 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.006323  8773 tablet_bootstrap.cc:492] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.014158  8774 tablet_bootstrap.cc:492] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.015286  8584 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.016897  8774 ts_tablet_manager.cc:1403] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.029s	user 0.014s	sys 0.004s
I20260709 13:00:58.020390  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:58.021536  8817 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.022075  8817 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:58.022277  8774 raft_consensus.cc:359] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.023095  8773 tablet_bootstrap.cc:654] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.022972  8817 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.022980  8774 raft_consensus.cc:385] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.026440  8659 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.027395  8774 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.028505  8774 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.031268  8774 ts_tablet_manager.cc:1434] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.003s
I20260709 13:00:58.034920  8774 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.041568  8481 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:58.053292  8774 tablet_bootstrap.cc:654] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.063747  8775 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.064404  8775 ts_tablet_manager.cc:1403] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.087s	user 0.019s	sys 0.016s
I20260709 13:00:58.067579  8775 raft_consensus.cc:359] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.068598  8775 raft_consensus.cc:385] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.068969  8775 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.069921  8775 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.069224  8480 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:58.071653  8482 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:58.072588  8775 ts_tablet_manager.cc:1434] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:00:58.073783  8775 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.079344  8775 tablet_bootstrap.cc:654] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.089653  8773 tablet_bootstrap.cc:492] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.089756  8774 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.090106  8775 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.090350  8774 ts_tablet_manager.cc:1403] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.056s	user 0.024s	sys 0.008s
I20260709 13:00:58.090292  8773 ts_tablet_manager.cc:1403] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.084s	user 0.026s	sys 0.014s
I20260709 13:00:58.090857  8775 ts_tablet_manager.cc:1403] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.007s
I20260709 13:00:58.093472  8774 raft_consensus.cc:359] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.094570  8797 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.093973  8775 raft_consensus.cc:359] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.095265  8775 raft_consensus.cc:385] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.095124  8797 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.094223  8774 raft_consensus.cc:385] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.096068  8774 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.095760  8775 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.096966  8774 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.097221  8775 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.099547  8775 ts_tablet_manager.cc:1434] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:00:58.100220  8774 ts_tablet_manager.cc:1434] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:58.099982  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.100361  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.101315  8775 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
W20260709 13:00:58.101890  8625 leader_election.cc:343] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:58.102408  8622 leader_election.cc:343] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.102890  8622 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:00:58.103142  8834 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.103802  8828 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.103659  8834 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.104564  8797 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.093827  8773 raft_consensus.cc:359] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.108455  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.108868  8773 raft_consensus.cc:385] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.109004  8775 tablet_bootstrap.cc:654] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.109278  8773 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.110217  8773 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.111287  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:00:58.112433  8549 leader_election.cc:343] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.113073  8834 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.113883  8774 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: Bootstrap starting.
W20260709 13:00:58.118078  8550 leader_election.cc:343] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.118839  8550 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.120229  8774 tablet_bootstrap.cc:654] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.120445  8834 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.120484  8817 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.122318  8773 ts_tablet_manager.cc:1434] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.003s
I20260709 13:00:58.122004  8817 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.125125  8775 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.126226  8775 ts_tablet_manager.cc:1403] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.010s
I20260709 13:00:58.126948  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.128057  8817 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.129232  8775 raft_consensus.cc:359] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.130024  8775 raft_consensus.cc:385] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.130347  8775 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
W20260709 13:00:58.129084  8696 leader_election.cc:343] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.131454  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.131283  8775 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.132102  8659 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:58.133766  8773 tablet_bootstrap.cc:492] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.133630  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:58.135089  8796 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.135524  8796 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.135893  8796 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.136960  8775 ts_tablet_manager.cc:1434] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:00:58.138064  8775 tablet_bootstrap.cc:492] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.143486  8773 tablet_bootstrap.cc:654] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.144327  8775 tablet_bootstrap.cc:654] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.157166  8773 tablet_bootstrap.cc:492] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.157912  8773 ts_tablet_manager.cc:1403] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.007s
I20260709 13:00:58.162395  8773 raft_consensus.cc:359] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.163455  8773 raft_consensus.cc:385] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.163841  8773 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.164146  8796 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.165081  8773 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.169051  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:00:58.171417  8696 leader_election.cc:343] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.174695  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.175719  8659 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.180534  8773 ts_tablet_manager.cc:1434] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260709 13:00:58.187747  8659 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.189776  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:58.191498  8817 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.192341  8773 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.192255  8817 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:58.193675  8817 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:58.195915  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:58.199048  8796 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.200698  8773 tablet_bootstrap.cc:654] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.214848  8773 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.215569  8773 ts_tablet_manager.cc:1403] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.007s
I20260709 13:00:58.215131  8481 catalog_manager.cc:5697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:58.222173  8773 raft_consensus.cc:359] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:58.222978  8773 raft_consensus.cc:385] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.223250  8774 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.223309  8773 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.223860  8774 ts_tablet_manager.cc:1403] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.110s	user 0.047s	sys 0.045s
I20260709 13:00:58.224426  8773 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:58.226210  8775 tablet_bootstrap.cc:492] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.226611  8773 ts_tablet_manager.cc:1434] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.002s
I20260709 13:00:58.226706  8775 ts_tablet_manager.cc:1403] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.089s	user 0.035s	sys 0.040s
I20260709 13:00:58.226832  8774 raft_consensus.cc:359] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.227691  8774 raft_consensus.cc:385] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.227986  8774 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.228087  8773 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.229815  8775 raft_consensus.cc:359] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.229662  8774 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.230487  8775 raft_consensus.cc:385] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.230816  8775 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.232112  8775 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.235417  8773 tablet_bootstrap.cc:654] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.235648  8774 ts_tablet_manager.cc:1434] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:00:58.235652  8775 ts_tablet_manager.cc:1434] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:00:58.236860  8775 tablet_bootstrap.cc:492] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.236860  8774 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.245353  8775 tablet_bootstrap.cc:654] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.245453  8774 tablet_bootstrap.cc:654] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.247532  8797 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.247998  8797 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } }
I20260709 13:00:58.250078  8797 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:00:58.252056  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.252882  8732 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:00:58.254277  8796 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.255018  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.254954  8796 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.256795  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:00:58.258368  8828 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.260032  8796 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.258227  8817 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.261408  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.261727  8817 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:58.263435  8698 leader_election.cc:343] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.263926  8817 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.265319  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.265515  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.265812  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.266707  8659 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
W20260709 13:00:58.267050  8696 leader_election.cc:343] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.267563  8696 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:00:58.268800  8696 leader_election.cc:343] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.269418  8698 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:58.270430  8796 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.270932  8817 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.273566  8817 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.273806  8775 tablet_bootstrap.cc:492] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.273967  8817 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.274801  8774 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.274849  8775 ts_tablet_manager.cc:1403] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.038s	user 0.016s	sys 0.013s
I20260709 13:00:58.275355  8774 ts_tablet_manager.cc:1403] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.039s	user 0.015s	sys 0.006s
I20260709 13:00:58.247082  8773 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.277748  8773 ts_tablet_manager.cc:1403] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.050s	user 0.010s	sys 0.008s
I20260709 13:00:58.277611  8775 raft_consensus.cc:359] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.277752  8774 raft_consensus.cc:359] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.278321  8775 raft_consensus.cc:385] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.278625  8774 raft_consensus.cc:385] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.278712  8775 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.278944  8774 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.279480  8828 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.279691  8774 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.279585  8775 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.280584  8773 raft_consensus.cc:359] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.281265  8773 raft_consensus.cc:385] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.281543  8773 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.282191  8773 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.284210  8773 ts_tablet_manager.cc:1434] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:00:58.285077  8774 ts_tablet_manager.cc:1434] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:58.286073  8774 tablet_bootstrap.cc:492] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.286826  8773 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.289296  8817 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.292318  8775 ts_tablet_manager.cc:1434] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 13:00:58.292508  8774 tablet_bootstrap.cc:654] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.292631  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:00:58.293891  8696 leader_election.cc:343] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.293905  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.295392  8659 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.296722  8775 tablet_bootstrap.cc:492] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.304388  8775 tablet_bootstrap.cc:654] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.307485  8659 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.308964  8698 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:58.309928  8817 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.310003  8796 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.310745  8796 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:58.311930  8796 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.310765  8773 tablet_bootstrap.cc:654] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.314230  8774 tablet_bootstrap.cc:492] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.314721  8774 ts_tablet_manager.cc:1403] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.029s	user 0.017s	sys 0.000s
I20260709 13:00:58.315762  8775 tablet_bootstrap.cc:492] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.316330  8775 ts_tablet_manager.cc:1403] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.008s
I20260709 13:00:58.319020  8774 raft_consensus.cc:359] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.323987  8774 raft_consensus.cc:385] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.325603  8774 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.326550  8774 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.328753  8774 ts_tablet_manager.cc:1434] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.001s
I20260709 13:00:58.328909  8775 raft_consensus.cc:359] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.329581  8775 raft_consensus.cc:385] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.329655  8774 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.329910  8775 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.330556  8775 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.332442  8775 ts_tablet_manager.cc:1434] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 13:00:58.333459  8775 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.333274  8482 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:58.335945  8773 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.336400  8773 ts_tablet_manager.cc:1403] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.050s	user 0.011s	sys 0.007s
I20260709 13:00:58.336409  8834 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.336985  8834 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.339154  8773 raft_consensus.cc:359] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.339881  8773 raft_consensus.cc:385] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.340138  8773 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.340298  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.340912  8773 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:58.341850  8549 leader_election.cc:343] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.342088  8775 tablet_bootstrap.cc:654] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.343048  8827 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.343523  8827 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.344415  8834 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.346005  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.346733  8732 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:58.348095  8774 tablet_bootstrap.cc:654] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.347985  8550 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.349218  8834 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.349629  8834 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.350068  8834 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.346128  8827 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.354630  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.355578  8732 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:00:58.356769  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:58.357038  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.360733  8773 ts_tablet_manager.cc:1434] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 13:00:58.362146  8773 tablet_bootstrap.cc:492] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.362260  8827 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:00:58.362465  8622 leader_election.cc:343] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.362725  8827 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.363073  8827 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.365513  8833 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.365996  8833 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.368429  8833 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.370100  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.371163  8659 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:00:58.371551  8773 tablet_bootstrap.cc:654] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.373123  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.374678  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.375402  8774 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.375955  8774 ts_tablet_manager.cc:1403] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.046s	user 0.016s	sys 0.000s
I20260709 13:00:58.376041  8833 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.378747  8774 raft_consensus.cc:359] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.379611  8774 raft_consensus.cc:385] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.379935  8774 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.380792  8774 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.383342  8774 ts_tablet_manager.cc:1434] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:58.384321  8774 tablet_bootstrap.cc:492] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.391397  8833 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.392400  8774 tablet_bootstrap.cc:654] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.392283  8834 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.395222  8833 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.395709  8833 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.397130  8773 tablet_bootstrap.cc:492] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.397814  8773 ts_tablet_manager.cc:1403] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.036s	user 0.018s	sys 0.003s
I20260709 13:00:58.398708  8833 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.400019  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.400830  8732 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:58.402551  8550 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:58.403450  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.402963  8773 raft_consensus.cc:359] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.404135  8659 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:58.403982  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.404515  8773 raft_consensus.cc:385] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.403801  8834 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.405326  8773 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
W20260709 13:00:58.406338  8549 leader_election.cc:343] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.407130  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:58.407513  8773 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.408136  8733 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.408408  8833 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.410557  8773 ts_tablet_manager.cc:1434] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.002s
I20260709 13:00:58.411701  8773 tablet_bootstrap.cc:492] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.408758  8833 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.409366  8827 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.414342  8833 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.417356  8827 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.417565  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:58.418176  8732 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.419195  8733 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:00:58.420063  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:58.421356  8550 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:00:58.422602  8622 leader_election.cc:343] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.423028  8775 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.423494  8781 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.423983  8781 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:00:58.425139  8781 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.427704  8773 tablet_bootstrap.cc:654] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.438028  8733 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.444573  8732 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:00:58.443938  8480 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:58.445727  8773 tablet_bootstrap.cc:492] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.446301  8773 ts_tablet_manager.cc:1403] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.035s	user 0.011s	sys 0.007s
I20260709 13:00:58.446197  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:58.447469  8781 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:58.423532  8775 ts_tablet_manager.cc:1403] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.090s	user 0.044s	sys 0.029s
I20260709 13:00:58.448262  8827 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.449389  8827 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:00:58.446602  8859 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:00:58.450378  8827 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.451431  8775 raft_consensus.cc:359] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.452159  8775 raft_consensus.cc:385] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.452426  8775 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.453107  8775 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.455986  8773 raft_consensus.cc:359] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.456825  8773 raft_consensus.cc:385] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.455761  8833 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.457165  8773 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.457885  8775 ts_tablet_manager.cc:1434] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:00:58.457895  8773 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.458966  8775 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.460989  8773 ts_tablet_manager.cc:1434] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:00:58.460886  8833 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.464215  8773 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.465322  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.465994  8658 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.466053  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:58.470258  8731 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.470575  8774 tablet_bootstrap.cc:492] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.471083  8774 ts_tablet_manager.cc:1403] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.087s	user 0.015s	sys 0.010s
I20260709 13:00:58.477502  8833 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.478025  8833 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.481005  8833 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.480970  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.483588  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.483690  8731 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
W20260709 13:00:58.485421  8549 leader_election.cc:343] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.489033  8774 raft_consensus.cc:359] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:58.487500  8550 leader_election.cc:343] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.490168  8796 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.491010  8796 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.493891  8796 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.490020  8774 raft_consensus.cc:385] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.495615  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.496419  8833 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.496981  8550 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:58.495462  8774 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.494486  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.496565  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.498575  8583 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:58.498766  8659 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:58.497905  8833 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.499658  8833 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:00:58.500123  8859 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:58.498175  8774 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.501451  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:58.500546  8833 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.502564  8480 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:58.503978  8796 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.506544  8773 tablet_bootstrap.cc:654] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.506297  8796 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.507004  8796 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.509377  8774 ts_tablet_manager.cc:1434] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 13:00:58.510309  8774 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.517045  8817 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.517619  8817 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.521121  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.521806  8583 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:58.524087  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:58.500527  8658 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:00:58.528398  8775 tablet_bootstrap.cc:654] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.528290  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.529912  8814 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.530793  8814 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.531520  8814 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.532495  8796 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.533660  8834 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.534761  8796 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.534308  8834 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.525837  8817 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.548645  8834 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:00:58.537570  8698 leader_election.cc:343] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.555967  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.556291  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:58.556970  8733 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:58.557241  8583 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.572047  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.572736  8657 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:58.573688  8833 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:58.568868  8814 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.576174  8549 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:58.581979  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.590145  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.590946  8657 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.585186  8781 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.584553  8814 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.587726  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:58.602514  8584 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.595077  8781 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.606283  8781 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:58.592307  8698 leader_election.cc:343] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.616353  8781 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.621582  8781 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.624115  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.624987  8659 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.625850  8781 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.626403  8781 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.629715  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.630916  8814 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:00:58.640513  8583 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.641788  8781 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.643620  8584 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.643877  8479 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:58.650979  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:00:58.652809  8549 leader_election.cc:343] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.660930  8774 tablet_bootstrap.cc:654] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.663842  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.664278  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:58.665726  8817 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.669770  8817 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:58.670621  8817 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.666417  8814 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.674422  8814 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:58.674695  8659 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:00:58.675285  8814 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.664973  8733 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.687958  8549 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:58.669247  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.691990  8781 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:00:58.694648  8550 leader_election.cc:343] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.700927  8781 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:00:58.700974  8550 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.706693  8869 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.709461  8775 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.709884  8775 ts_tablet_manager.cc:1403] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.251s	user 0.020s	sys 0.008s
I20260709 13:00:58.701925  8781 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.708482  8796 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.718119  8796 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.719971  8796 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.722306  8773 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.722692  8773 ts_tablet_manager.cc:1403] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.260s	user 0.027s	sys 0.002s
I20260709 13:00:58.710882  8827 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.722036  8657 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
W20260709 13:00:58.731724  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:58.728533  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.724893  8827 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:58.735224  8696 leader_election.cc:343] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.738178  8827 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.739210  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:58.739413  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.739931  8582 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
W20260709 13:00:58.740703  8625 leader_election.cc:343] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.739236  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.742249  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:00:58.743295  8827 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.743758  8827 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.742955  8877 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 13:00:58.745610  8733 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:00:58.745601  8827 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.751995  8774 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.752382  8774 ts_tablet_manager.cc:1403] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.242s	user 0.020s	sys 0.013s
I20260709 13:00:58.753350  8773 raft_consensus.cc:359] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.754112  8773 raft_consensus.cc:385] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.754417  8773 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.754249  8775 raft_consensus.cc:359] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.755084  8775 raft_consensus.cc:385] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.755435  8775 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.756438  8775 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.761896  8773 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.763885  8773 ts_tablet_manager.cc:1434] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.041s	user 0.002s	sys 0.004s
I20260709 13:00:58.764688  8827 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.766482  8482 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:58.767824  8482 catalog_manager.cc:5697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:58.770445  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:58.771077  8584 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.770804  8877 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:58.774641  8775 ts_tablet_manager.cc:1434] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.064s	user 0.008s	sys 0.000s
I20260709 13:00:58.775611  8775 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.778982  8827 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:58.790308  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:00:58.794188  8625 leader_election.cc:343] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.802284  8584 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:00:58.800216  8774 raft_consensus.cc:359] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.803345  8774 raft_consensus.cc:385] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.803648  8774 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.804637  8774 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.806018  8654 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:58.806191  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:00:58.807377  8698 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:58.808197  8797 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.808671  8796 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.808673  8797 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:00:58.809063  8773 tablet_bootstrap.cc:492] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.809115  8796 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.809651  8796 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.809928  8797 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.810652  8775 tablet_bootstrap.cc:654] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.817081  8773 tablet_bootstrap.cc:654] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.817519  8774 ts_tablet_manager.cc:1434] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.065s	user 0.008s	sys 0.000s
I20260709 13:00:58.818565  8774 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.825241  8480 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:58.826457  8774 tablet_bootstrap.cc:654] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.827345  8775 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.830808  8773 tablet_bootstrap.cc:492] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.832096  8796 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.833155  8775 ts_tablet_manager.cc:1403] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.058s	user 0.011s	sys 0.000s
I20260709 13:00:58.833483  8773 ts_tablet_manager.cc:1403] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.008s
I20260709 13:00:58.835625  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:58.835929  8775 raft_consensus.cc:359] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.836187  8773 raft_consensus.cc:359] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.836625  8775 raft_consensus.cc:385] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.837035  8775 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
W20260709 13:00:58.837090  8696 leader_election.cc:343] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.838107  8775 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.839427  8796 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.840009  8773 raft_consensus.cc:385] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.840236  8775 ts_tablet_manager.cc:1434] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:00:58.840353  8773 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.841168  8775 tablet_bootstrap.cc:492] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.841207  8773 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.843448  8773 ts_tablet_manager.cc:1434] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260709 13:00:58.844276  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.844985  8655 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.847877  8775 tablet_bootstrap.cc:654] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.856714  8655 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.858417  8698 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:58.859967  8775 tablet_bootstrap.cc:492] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.860569  8775 ts_tablet_manager.cc:1403] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.008s
I20260709 13:00:58.860754  8877 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.861500  8877 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.863094  8775 raft_consensus.cc:359] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.863648  8775 raft_consensus.cc:385] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.863874  8775 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.864638  8775 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.865259  8796 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:58.865384  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.866056  8796 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:58.867231  8796 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.869170  8890 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:00:58.871189  8696 leader_election.cc:343] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.872062  8877 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.873624  8773 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:58.874114  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.874630  8797 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:58.879335  8775 ts_tablet_manager.cc:1434] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.001s
I20260709 13:00:58.880440  8775 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.884490  8774 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.885159  8774 ts_tablet_manager.cc:1403] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.067s	user 0.039s	sys 0.010s
W20260709 13:00:58.886978  8698 leader_election.cc:343] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.887511  8698 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:58.889488  8890 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:58.890067  8773 tablet_bootstrap.cc:654] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.887522  8479 catalog_manager.cc:5697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:58.894227  8775 tablet_bootstrap.cc:654] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.896287  8774 raft_consensus.cc:359] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.896644  8796 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:58.897323  8796 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.897202  8774 raft_consensus.cc:385] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.899813  8774 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.903327  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:58.904436  8584 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:58.906302  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:58.907569  8890 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:58.908002  8890 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:58.908454  8890 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.903518  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:58.903424  8774 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:58.911514  8698 leader_election.cc:343] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.919195  8774 ts_tablet_manager.cc:1434] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.034s	user 0.002s	sys 0.005s
I20260709 13:00:58.920363  8774 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.904469  8796 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.926792  8775 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:58.927287  8775 ts_tablet_manager.cc:1403] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.008s
I20260709 13:00:58.929999  8775 raft_consensus.cc:359] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.930712  8775 raft_consensus.cc:385] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.930990  8775 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.934095  8773 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:58.934666  8773 ts_tablet_manager.cc:1403] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.061s	user 0.021s	sys 0.022s
I20260709 13:00:58.936282  8774 tablet_bootstrap.cc:654] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.937489  8773 raft_consensus.cc:359] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.938254  8773 raft_consensus.cc:385] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.938591  8773 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.939448  8773 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.941656  8773 ts_tablet_manager.cc:1434] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:58.942689  8773 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
W20260709 13:00:58.945834  8622 consensus_peers.cc:597] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:58.938607  8775 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.948269  8775 ts_tablet_manager.cc:1434] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 13:00:58.948057  8877 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:58.951493  8781 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:58.954743  8775 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:58.918226  8890 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.960366  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:58.961289  8775 tablet_bootstrap.cc:654] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:58.961747  8698 leader_election.cc:343] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:58.962545  8890 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:58.969904  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:58.970889  8582 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:58.951917  8774 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:58.953250  8773 tablet_bootstrap.cc:654] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:58.977035  8774 ts_tablet_manager.cc:1403] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.057s	user 0.008s	sys 0.004s
I20260709 13:00:58.979970  8774 raft_consensus.cc:359] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.980667  8774 raft_consensus.cc:385] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:58.980938  8774 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:58.981688  8774 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:58.984408  8774 ts_tablet_manager.cc:1434] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20260709 13:00:58.985234  8482 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:58.989627  8869 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:58.985423  8774 tablet_bootstrap.cc:492] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:58.996577  8582 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:58.998476  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.005612  8877 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:59.006467  8773 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:59.006399  8877 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:59.007565  8877 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.012110  8773 ts_tablet_manager.cc:1403] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.070s	user 0.019s	sys 0.005s
I20260709 13:00:59.012621  8774 tablet_bootstrap.cc:654] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.015210  8773 raft_consensus.cc:359] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.016453  8773 raft_consensus.cc:385] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.016821  8773 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.018116  8775 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:59.018831  8775 ts_tablet_manager.cc:1403] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.064s	user 0.011s	sys 0.021s
I20260709 13:00:59.018175  8773 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.021865  8775 raft_consensus.cc:359] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.022166  8833 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:00:59.025712  8773 ts_tablet_manager.cc:1434] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:00:59.027197  8773 tablet_bootstrap.cc:492] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:59.022552  8775 raft_consensus.cc:385] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.028013  8775 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.029592  8775 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.034041  8773 tablet_bootstrap.cc:654] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.050436  8773 tablet_bootstrap.cc:492] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:59.051011  8773 ts_tablet_manager.cc:1403] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.006s
I20260709 13:00:59.051882  8655 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.052057  8774 tablet_bootstrap.cc:492] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:59.052879  8774 ts_tablet_manager.cc:1403] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.068s	user 0.009s	sys 0.010s
I20260709 13:00:59.058075  8774 raft_consensus.cc:359] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.058866  8774 raft_consensus.cc:385] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.059201  8774 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.059947  8859 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.059898  8774 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.060498  8859 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.062891  8774 ts_tablet_manager.cc:1434] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:00:59.063277  8859 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.063993  8774 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:59.049029  8775 ts_tablet_manager.cc:1434] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.000s
I20260709 13:00:59.065506  8775 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:59.066314  8877 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.066821  8877 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.068075  8773 raft_consensus.cc:359] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.068861  8773 raft_consensus.cc:385] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.069152  8877 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.069168  8773 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.072784  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:59.073348  8775 tablet_bootstrap.cc:654] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.073565  8584 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:59.073657  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:59.072700  8773 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.075084  8654 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:00:59.076598  8773 ts_tablet_manager.cc:1434] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.025s	user 0.002s	sys 0.003s
I20260709 13:00:59.076833  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:59.077729  8773 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:59.077859  8877 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:59.078294  8877 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.078146  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:59.078810  8877 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.074352  8859 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:59.081678  8774 tablet_bootstrap.cc:654] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.083858  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.085631  8781 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.086288  8781 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.092200  8859 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.093427  8773 tablet_bootstrap.cc:654] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.095011  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:59.099442  8549 leader_election.cc:343] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.102517  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:59.103555  8731 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:59.089092  8781 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.106329  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.107748  8781 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:59.108126  8781 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.108425  8781 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.109546  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:59.110975  8698 leader_election.cc:343] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.096297  8481 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.115362  8655 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.119879  8877 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.120781  8834 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:59.123282  8890 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:59.117380  8859 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:59.124881  8877 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.123468  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:59.127017  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:00:59.128319  8698 leader_election.cc:343] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.129997  8584 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.164863  8781 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.189628  8775 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:59.190192  8775 ts_tablet_manager.cc:1403] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.125s	user 0.037s	sys 0.070s
I20260709 13:00:59.193581  8584 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:59.196310  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:59.196468  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.196923  8731 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.197700  8890 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:59.198190  8890 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:59.200894  8773 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:59.201391  8773 ts_tablet_manager.cc:1403] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.124s	user 0.032s	sys 0.052s
I20260709 13:00:59.206730  8890 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.194970  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:59.209618  8827 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:59.209775  8549 leader_election.cc:343] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.210157  8827 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.214643  8827 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.218461  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:00:59.220570  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:59.220333  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:59.224121  8775 raft_consensus.cc:359] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:59.224972  8622 leader_election.cc:343] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.226220  8877 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.226737  8877 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.229737  8877 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.230333  8877 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:59.225140  8775 raft_consensus.cc:385] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.232201  8775 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.194722  8781 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.231374  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:00:59.238227  8625 leader_election.cc:343] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.238895  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:00:59.243728  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:59.244766  8659 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:59.244992  8797 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:59.234789  8775 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:59.252300  8696 leader_election.cc:343] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.254726  8817 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.255471  8817 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.255983  8698 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:00:59.256237  8781 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:59.257211  8796 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:59.257632  8796 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.257968  8796 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.258464  8817 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.261438  8731 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:00:59.263109  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.264653  8781 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:59.265424  8781 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:00:59.264791  8582 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.234042  8877 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260709 13:00:59.266566  8781 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.271518  8775 ts_tablet_manager.cc:1434] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.081s	user 0.008s	sys 0.001s
I20260709 13:00:59.272238  8775 tablet_bootstrap.cc:492] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:59.274091  8797 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:59.274520  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:59.275158  8583 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:59.266647  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:59.275768  8859 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.276420  8859 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.282461  8859 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.283535  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:59.286947  8549 leader_election.cc:343] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.286903  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:59.293252  8731 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:00:59.307201  8774 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:59.307907  8774 ts_tablet_manager.cc:1403] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.244s	user 0.026s	sys 0.010s
I20260709 13:00:59.293308  8696 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:59.307876  8773 raft_consensus.cc:359] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.313611  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.314774  8859 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:59.315138  8859 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.313293  8773 raft_consensus.cc:385] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.315869  8773 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.316932  8773 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:00:59.317790  8625 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:59.318998  8773 ts_tablet_manager.cc:1434] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.117s	user 0.007s	sys 0.000s
I20260709 13:00:59.315510  8859 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.319681  8774 raft_consensus.cc:359] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.320362  8774 raft_consensus.cc:385] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.320638  8774 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.321290  8774 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.333496  8775 tablet_bootstrap.cc:654] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.336614  8774 ts_tablet_manager.cc:1434] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.028s	user 0.002s	sys 0.005s
I20260709 13:00:59.338806  8774 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:59.342514  8773 tablet_bootstrap.cc:492] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:59.345333  8814 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:59.354421  8814 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:59.355481  8656 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:59.357242  8814 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:59.335987  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46945us
I20260709 13:00:59.361639  8869 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.364776  8869 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.372042  8869 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.373663  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:59.374629  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:59.375180  8550 leader_election.cc:343] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:59.376041  8549 leader_election.cc:343] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.375802  8814 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.376458  8549 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.377543  8869 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:59.380735  8814 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.382182  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:59.382778  8654 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.386787  8859 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.402595  8877 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:59.410722  8654 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:59.412935  8482 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.416239  8774 tablet_bootstrap.cc:654] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.397132  8796 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.423954  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:59.421331  8796 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.427726  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:59.428325  8582 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.435657  8814 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:59.424533  8817 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 13:00:59.459496  8814 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:59.460394  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:00:59.461040  8732 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.460768  8814 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.486645  8797 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:59.556421  8917 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:00:59.557947  8859 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:00:59.560389  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 123427us
I20260709 13:00:59.558869  8797 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.575574  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:59.576385  8658 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.618340  8481 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:59.631194  8775 tablet_bootstrap.cc:492] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:59.631832  8775 ts_tablet_manager.cc:1403] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.360s	user 0.012s	sys 0.007s
I20260709 13:00:59.635237  8775 raft_consensus.cc:359] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.636157  8775 raft_consensus.cc:385] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.636489  8775 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.637358  8775 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.639338  8582 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:59.641669  8797 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.646111  8775 ts_tablet_manager.cc:1434] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 13:00:59.647351  8775 tablet_bootstrap.cc:492] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:59.647406  8658 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:59.655162  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:59.649659  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:59.656388  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:00:59.656977  8584 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:00:59.679060  8773 tablet_bootstrap.cc:654] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.683490  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:59.691633  8732 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:00:59.693135  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:59.694141  8859 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:59.695581  8859 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:00:59.696462  8698 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:00:59.696425  8859 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.707355  8654 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.699700  8921 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:59.709789  8921 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:59.710630  8921 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.712716  8481 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.720885  8774 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:59.721578  8774 ts_tablet_manager.cc:1403] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.383s	user 0.010s	sys 0.012s
I20260709 13:00:59.721647  8775 tablet_bootstrap.cc:654] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:59.724710  8698 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:59.739504  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:00:59.743432  8927 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:59.743898  8774 raft_consensus.cc:359] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.744834  8774 raft_consensus.cc:385] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.745106  8774 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.746223  8774 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.750304  8774 ts_tablet_manager.cc:1434] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260709 13:00:59.751224  8774 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:59.752112  8877 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:00:59.753151  8696 leader_election.cc:343] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.753469  8877 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:59.759877  8781 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:59.763216  8773 tablet_bootstrap.cc:492] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:59.763958  8773 ts_tablet_manager.cc:1403] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.422s	user 0.014s	sys 0.026s
I20260709 13:00:59.767907  8814 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:00:59.777415  8481 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.788235  8774 tablet_bootstrap.cc:654] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:59.791766  8549 leader_election.cc:343] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.795739  8773 raft_consensus.cc:359] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.796309  8733 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.796558  8773 raft_consensus.cc:385] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.809689  8773 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.810616  8773 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.816731  8773 ts_tablet_manager.cc:1434] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.001s
I20260709 13:00:59.819020  8774 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:59.819535  8774 ts_tablet_manager.cc:1403] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.068s	user 0.009s	sys 0.002s
I20260709 13:00:59.822188  8774 raft_consensus.cc:359] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.822904  8774 raft_consensus.cc:385] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.823215  8774 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.825542  8773 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:59.828406  8775 tablet_bootstrap.cc:492] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:59.828967  8775 ts_tablet_manager.cc:1403] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.182s	user 0.009s	sys 0.008s
W20260709 13:00:59.829712  8549 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:59.832259  8773 tablet_bootstrap.cc:654] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.832667  8877 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:59.823936  8774 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.832975  8482 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:59.835402  8774 ts_tablet_manager.cc:1434] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.003s
I20260709 13:00:59.836395  8774 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:59.838745  8775 raft_consensus.cc:359] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.839429  8775 raft_consensus.cc:385] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.839707  8775 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.840408  8775 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.842454  8774 tablet_bootstrap.cc:654] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.842792  8775 ts_tablet_manager.cc:1434] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 13:00:59.843705  8775 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:59.846850  8814 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:00:59.847473  8814 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.849923  8775 tablet_bootstrap.cc:654] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.852488  8814 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.852507  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:59.867755  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:59.869805  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.871392  8890 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:00:59.900313  8773 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:00:59.900840  8773 ts_tablet_manager.cc:1403] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.076s	user 0.014s	sys 0.024s
I20260709 13:00:59.903252  8775 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:59.903825  8775 ts_tablet_manager.cc:1403] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.060s	user 0.020s	sys 0.004s
I20260709 13:00:59.903870  8773 raft_consensus.cc:359] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.904695  8773 raft_consensus.cc:385] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.904980  8773 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.905659  8773 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.907276  8775 raft_consensus.cc:359] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.907940  8775 raft_consensus.cc:385] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.908198  8775 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.908324  8774 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:59.908921  8774 ts_tablet_manager.cc:1403] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.073s	user 0.023s	sys 0.012s
I20260709 13:00:59.909044  8775 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.911525  8775 ts_tablet_manager.cc:1434] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:00:59.913085  8773 ts_tablet_manager.cc:1434] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.004s
I20260709 13:00:59.914057  8773 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:00:59.914909  8775 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:59.916934  8774 raft_consensus.cc:359] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.917744  8774 raft_consensus.cc:385] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.918076  8774 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.918834  8774 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.921290  8774 ts_tablet_manager.cc:1434] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.001s
I20260709 13:00:59.921658  8775 tablet_bootstrap.cc:654] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.922395  8774 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:59.932170  8774 tablet_bootstrap.cc:654] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.935554  8773 tablet_bootstrap.cc:654] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.939293  8921 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.939958  8921 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.945067  8890 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:59.948833  8859 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:59.949584  8859 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.953676  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:00:59.954200  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:00:59.955154  8550 leader_election.cc:343] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.955125  8921 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:00:59.955996  8549 leader_election.cc:343] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.956651  8549 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.958068  8781 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:59.959770  8859 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:00:59.959970  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:00:59.959636  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:00:59.956583  8921 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:59.960664  8581 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:00:59.962110  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:00:59.963222  8921 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:59.963797  8921 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:59.963893  8698 leader_election.cc:343] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.964455  8921 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.969106  8774 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:00:59.969672  8774 ts_tablet_manager.cc:1403] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.047s	user 0.024s	sys 0.012s
I20260709 13:00:59.971385  8775 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:00:59.971882  8775 ts_tablet_manager.cc:1403] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.057s	user 0.016s	sys 0.027s
I20260709 13:00:59.972656  8774 raft_consensus.cc:359] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.972797  8921 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.973388  8774 raft_consensus.cc:385] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.973717  8774 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.974472  8774 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.974648  8775 raft_consensus.cc:359] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.975644  8775 raft_consensus.cc:385] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:59.975968  8775 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:00:59.976101  8774 ts_tablet_manager.cc:1434] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:00:59.975980  8921 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:00:59.977708  8774 tablet_bootstrap.cc:492] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:00:59.977203  8775 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:00:59.979070  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:00:59.979663  8581 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:59.982334  8775 ts_tablet_manager.cc:1434] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:00:59.982391  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:00:59.983868  8698 leader_election.cc:343] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:59.985304  8775 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:00:59.989797  8581 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:00:59.991345  8775 tablet_bootstrap.cc:654] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.992795  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:00:59.994328  8890 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:59.995286  8890 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:00:59.997875  8774 tablet_bootstrap.cc:654] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:59.997437  8890 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.011984  8774 tablet_bootstrap.cc:492] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.012576  8774 ts_tablet_manager.cc:1403] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.035s	user 0.016s	sys 0.004s
I20260709 13:01:00.015179  8774 raft_consensus.cc:359] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.016011  8774 raft_consensus.cc:385] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.016338  8774 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.020401  8479 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:00.021773  8774 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.023656  8781 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.024490  8774 ts_tablet_manager.cc:1434] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.001s
I20260709 13:01:00.025522  8774 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:00.028498  8781 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.030699  8781 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.031628  8581 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.031741  8774 tablet_bootstrap.cc:654] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.033164  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.049644  8773 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.050130  8773 ts_tablet_manager.cc:1403] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.136s	user 0.009s	sys 0.003s
I20260709 13:01:00.046461  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.052942  8655 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:01:00.054488  8775 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.054278  8549 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.054979  8775 ts_tablet_manager.cc:1403] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.070s	user 0.023s	sys 0.025s
I20260709 13:01:00.055462  8781 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.058164  8775 raft_consensus.cc:359] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.059425  8775 raft_consensus.cc:385] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.059736  8775 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.060391  8775 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.063361  8781 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.067866  8775 ts_tablet_manager.cc:1434] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:01:00.069219  8775 tablet_bootstrap.cc:492] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.076272  8781 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.076970  8781 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.077405  8774 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.078123  8774 ts_tablet_manager.cc:1403] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.053s	user 0.016s	sys 0.011s
I20260709 13:01:00.079209  8781 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.079922  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.080464  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:00.082023  8549 leader_election.cc:343] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:00.083250  8550 leader_election.cc:343] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.087152  8921 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.087741  8773 raft_consensus.cc:359] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.087926  8921 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.088395  8774 raft_consensus.cc:359] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.089311  8774 raft_consensus.cc:385] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.089323  8775 tablet_bootstrap.cc:654] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.089754  8774 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.091573  8921 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.088416  8773 raft_consensus.cc:385] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.092166  8774 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.092917  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.092360  8773 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.094776  8773 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:00.096673  8696 leader_election.cc:343] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.097384  8877 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.097940  8877 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.099244  8775 tablet_bootstrap.cc:492] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.099752  8775 ts_tablet_manager.cc:1403] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.031s	user 0.009s	sys 0.002s
I20260709 13:01:00.083684  8550 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.100404  8877 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.101825  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.102170  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.102588  8657 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:00.103816  8774 ts_tablet_manager.cc:1434] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.001s
I20260709 13:01:00.103542  8775 raft_consensus.cc:359] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.104342  8775 raft_consensus.cc:385] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.104516  8781 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.104698  8775 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.105376  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.105727  8775 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.106117  8581 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:00.108506  8775 ts_tablet_manager.cc:1434] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 13:01:00.109688  8775 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.110641  8773 ts_tablet_manager.cc:1434] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.060s	user 0.006s	sys 0.001s
I20260709 13:01:00.111967  8773 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:00.117674  8773 tablet_bootstrap.cc:654] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.104882  8774 tablet_bootstrap.cc:492] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:00.125264  8775 tablet_bootstrap.cc:654] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.125686  8774 tablet_bootstrap.cc:654] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
W20260709 13:01:00.128645  8698 leader_election.cc:343] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.129140  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.130581  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:00.134218  8877 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.134435  8921 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.134907  8921 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.135257  8921 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.136407  8775 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.136857  8775 ts_tablet_manager.cc:1403] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.007s
I20260709 13:01:00.139475  8775 raft_consensus.cc:359] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.140158  8775 raft_consensus.cc:385] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.140425  8775 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.141040  8775 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.146551  8775 ts_tablet_manager.cc:1434] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:01:00.147519  8775 tablet_bootstrap.cc:492] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.153163  8775 tablet_bootstrap.cc:654] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.153012  8921 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.164808  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:00.165704  8654 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.176122  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:00.177040  8581 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.183074  8654 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:00.183652  8921 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.189548  8775 tablet_bootstrap.cc:492] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.190006  8775 ts_tablet_manager.cc:1403] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.043s	user 0.026s	sys 0.010s
I20260709 13:01:00.192719  8775 raft_consensus.cc:359] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.193435  8775 raft_consensus.cc:385] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.193715  8775 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.194203  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:00.194309  8775 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.194965  8773 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.195447  8773 ts_tablet_manager.cc:1403] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.084s	user 0.004s	sys 0.004s
I20260709 13:01:00.195533  8890 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:00.195892  8775 ts_tablet_manager.cc:1434] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:01:00.196252  8890 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:00.196846  8775 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.197136  8890 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.198164  8773 raft_consensus.cc:359] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.198838  8773 raft_consensus.cc:385] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.199095  8773 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.200021  8773 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.203830  8775 tablet_bootstrap.cc:654] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.204385  8921 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:00.206976  8581 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:00.209753  8926 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.210291  8926 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.212273  8773 ts_tablet_manager.cc:1434] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.001s
I20260709 13:01:00.212477  8926 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.214435  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.215624  8775 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.216148  8775 ts_tablet_manager.cc:1403] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.011s
I20260709 13:01:00.220532  8926 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.220300  8921 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:00.221170  8926 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.224381  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.225317  8581 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:01:00.228111  8775 raft_consensus.cc:359] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.229161  8775 raft_consensus.cc:385] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.229813  8775 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.230057  8773 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:00.231894  8814 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.232450  8814 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.234162  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.234915  8732 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:00.235800  8814 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.236223  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:00.240468  8927 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.236989  8773 tablet_bootstrap.cc:654] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.240645  8622 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:00.240897  8927 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.241268  8927 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.237114  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.236575  8775 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:00.256768  8698 leader_election.cc:343] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.258234  8927 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.237802  8926 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.259330  8926 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.262313  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:00.263485  8732 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.265535  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.265667  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.266400  8581 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:00.258757  8775 ts_tablet_manager.cc:1434] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.003s
I20260709 13:01:00.274931  8965 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:00.276298  8696 leader_election.cc:343] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.281709  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.282691  8965 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.284833  8814 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.285485  8775 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.285537  8814 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.286242  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.286192  8814 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.287859  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.288998  8814 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.289201  8774 tablet_bootstrap.cc:492] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.289705  8733 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:00.290102  8965 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.279675  8926 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.291661  8927 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.292331  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.293005  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:00.293108  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:00.293675  8581 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.288931  8584 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:00.302485  8775 tablet_bootstrap.cc:654] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.285801  8773 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.303826  8773 ts_tablet_manager.cc:1403] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.074s	user 0.018s	sys 0.006s
I20260709 13:01:00.305335  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.306121  8654 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:00.307500  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:00.308363  8814 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.308830  8814 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.309348  8814 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.289726  8774 ts_tablet_manager.cc:1403] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.185s	user 0.048s	sys 0.020s
I20260709 13:01:00.308877  8966 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.317338  8966 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.318620  8926 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.323216  8926 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.322914  8966 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.318890  8775 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.324720  8775 ts_tablet_manager.cc:1403] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.039s	user 0.015s	sys 0.003s
I20260709 13:01:00.324980  8581 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:00.321170  8859 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:00.320598  8732 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
W20260709 13:01:00.336381  8549 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:00.335754  8966 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.326749  8774 raft_consensus.cc:359] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.330127  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.337196  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:00.325587  8654 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.338893  8774 raft_consensus.cc:385] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.339265  8774 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.339984  8774 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.330622  8926 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.332180  8479 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260709 13:01:00.331847  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:01:00.333537  8773 raft_consensus.cc:359] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.345381  8877 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.345902  8877 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.341318  8926 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:00.347976  8877 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.341784  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.348001  8926 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:00.349191  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.349913  8926 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.345453  8773 raft_consensus.cc:385] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.351094  8773 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.350973  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.351751  8584 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:00.348062  8774 ts_tablet_manager.cc:1434] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.002s
W20260709 13:01:00.352202  8698 leader_election.cc:343] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.351972  8773 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.353740  8774 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:00.353879  8696 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.354995  8877 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.351150  8622 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:00.355466  8877 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.355830  8877 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.335754  8814 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.362691  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:00.363188  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:00.363557  8731 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:01:00.363968  8581 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.365383  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:00.370674  8965 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.366544  8773 ts_tablet_manager.cc:1434] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.062s	user 0.005s	sys 0.001s
I20260709 13:01:00.370990  8966 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.372725  8773 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:00.372905  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:00.330442  8775 raft_consensus.cc:359] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.373734  8775 raft_consensus.cc:385] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.374047  8775 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.375769  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:00.376839  8657 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 1: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:01:00.379369  8775 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.380990  8814 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.382218  8966 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.385958  8581 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:00.386955  8877 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.387624  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.389232  8877 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.391202  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:01:00.392727  8698 leader_election.cc:343] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.396114  8775 ts_tablet_manager.cc:1434] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.071s	user 0.005s	sys 0.001s
I20260709 13:01:00.397244  8775 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.403643  8775 tablet_bootstrap.cc:654] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.410444  8775 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.410912  8775 ts_tablet_manager.cc:1403] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.004s
I20260709 13:01:00.410828  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:00.411540  8581 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.414130  8775 raft_consensus.cc:359] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.414848  8775 raft_consensus.cc:385] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.417062  8775 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.417726  8774 tablet_bootstrap.cc:654] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.417788  8775 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.419229  8773 tablet_bootstrap.cc:654] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.420317  8775 ts_tablet_manager.cc:1434] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:01:00.421602  8775 tablet_bootstrap.cc:492] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.422057  8877 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:00.422556  8877 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:00.423395  8877 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.422844  8481 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:00.428990  8581 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:00.430248  8696 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.431475  8890 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:00.432026  8890 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:00.432955  8775 tablet_bootstrap.cc:654] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.432901  8890 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.438283  8774 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.438800  8774 ts_tablet_manager.cc:1403] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.085s	user 0.017s	sys 0.006s
I20260709 13:01:00.439878  8773 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.440353  8773 ts_tablet_manager.cc:1403] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.068s	user 0.009s	sys 0.001s
I20260709 13:01:00.441702  8774 raft_consensus.cc:359] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.443339  8774 raft_consensus.cc:385] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.443713  8774 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.445472  8774 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.447934  8774 ts_tablet_manager.cc:1434] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:01:00.448935  8774 tablet_bootstrap.cc:492] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:00.455329  8774 tablet_bootstrap.cc:654] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.455693  8773 raft_consensus.cc:359] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.461890  8773 raft_consensus.cc:385] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.462349  8773 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.464569  8773 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.472512  8775 tablet_bootstrap.cc:492] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.473241  8775 ts_tablet_manager.cc:1403] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.052s	user 0.012s	sys 0.015s
I20260709 13:01:00.480654  8775 raft_consensus.cc:359] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.481580  8775 raft_consensus.cc:385] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.482120  8775 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.482920  8775 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.479079  8481 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:00.485579  8775 ts_tablet_manager.cc:1434] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 13:01:00.487020  8775 tablet_bootstrap.cc:492] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.487350  8773 ts_tablet_manager.cc:1434] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.047s	user 0.008s	sys 0.000s
I20260709 13:01:00.487105  8481 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:00.488922  8773 tablet_bootstrap.cc:492] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:00.492884  8774 tablet_bootstrap.cc:492] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.493424  8774 ts_tablet_manager.cc:1403] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.045s	user 0.031s	sys 0.011s
I20260709 13:01:00.494309  8775 tablet_bootstrap.cc:654] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.496309  8774 raft_consensus.cc:359] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.496997  8774 raft_consensus.cc:385] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.497298  8774 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.497975  8774 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.500151  8774 ts_tablet_manager.cc:1434] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:01:00.501821  8774 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:00.508318  8773 tablet_bootstrap.cc:654] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.512058  8774 tablet_bootstrap.cc:654] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.512861  8781 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.514128  8926 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.513761  8781 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.514662  8926 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.516757  8926 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.518021  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.518909  8733 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:00.520732  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.522015  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.523926  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:00.525189  8859 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.525715  8859 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.526170  8859 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.529876  8877 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.530401  8877 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.532512  8877 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.534458  8773 tablet_bootstrap.cc:492] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.534287  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.534785  8773 ts_tablet_manager.cc:1403] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.046s	user 0.013s	sys 0.005s
I20260709 13:01:00.535530  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:00.537153  8698 leader_election.cc:343] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.537045  8773 raft_consensus.cc:359] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.537784  8773 raft_consensus.cc:385] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.538188  8773 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.539090  8622 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:00.540313  8926 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.539157  8773 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.541592  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.553812  8890 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:00.554972  8773 ts_tablet_manager.cc:1434] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
W20260709 13:01:00.558699  8549 leader_election.cc:343] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:00.560029  8696 leader_election.cc:343] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.560612  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.562770  8890 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.564594  8781 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.572824  8773 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:00.582703  8773 tablet_bootstrap.cc:654] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.587491  8926 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.588367  8926 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.604153  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.605189  8733 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:00.606419  8877 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:00.607357  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:00.608392  8965 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.608563  8926 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.608837  8965 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.609483  8965 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.612576  8814 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.613106  8814 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.616169  8814 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.618322  8965 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.618557  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.622640  8965 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.623378  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.625499  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:00.625939  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:00.626292  8733 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.630268  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:00.630805  8622 leader_election.cc:343] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.581084  8859 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.647894  8859 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:00.632943  8696 leader_election.cc:343] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.649214  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:00.650141  8731 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:00.653018  8698 leader_election.cc:343] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:00.658067  8622 leader_election.cc:343] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.659968  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:00.663256  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:00.668817  8814 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.670694  8773 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.673473  8773 ts_tablet_manager.cc:1403] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.101s	user 0.011s	sys 0.005s
I20260709 13:01:00.674297  8774 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.674754  8774 ts_tablet_manager.cc:1403] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.173s	user 0.046s	sys 0.087s
I20260709 13:01:00.675274  8731 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:00.676813  8733 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:00.684949  8775 tablet_bootstrap.cc:492] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.685509  8775 ts_tablet_manager.cc:1403] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.199s	user 0.057s	sys 0.075s
I20260709 13:01:00.690795  8775 raft_consensus.cc:359] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.691810  8775 raft_consensus.cc:385] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.692129  8775 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.692790  8775 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.677505  8774 raft_consensus.cc:359] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.728091  8774 raft_consensus.cc:385] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.728415  8774 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.729102  8774 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.731238  8774 ts_tablet_manager.cc:1434] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.056s	user 0.007s	sys 0.000s
I20260709 13:01:00.730970  8773 raft_consensus.cc:359] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.731667  8773 raft_consensus.cc:385] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.732191  8919 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.733034  8775 ts_tablet_manager.cc:1434] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.047s	user 0.016s	sys 0.010s
I20260709 13:01:00.732767  8919 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.734000  8775 tablet_bootstrap.cc:492] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.734973  8919 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:00.731947  8773 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.737002  8773 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.738608  8774 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Bootstrap starting.
W20260709 13:01:00.741814  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44054.3us
I20260709 13:01:00.761061  8773 ts_tablet_manager.cc:1434] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.087s	user 0.007s	sys 0.000s
I20260709 13:01:00.763073  8773 tablet_bootstrap.cc:492] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:00.776401  8775 tablet_bootstrap.cc:654] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.777865  8773 tablet_bootstrap.cc:654] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.778626  8774 tablet_bootstrap.cc:654] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.678460  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:01:00.790903  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 112528us
I20260709 13:01:00.791613  8859 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:00.792135  8859 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:01:00.792335  8549 leader_election.cc:343] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.793056  8859 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.800215  8775 tablet_bootstrap.cc:492] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.801244  8775 ts_tablet_manager.cc:1403] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.067s	user 0.022s	sys 0.009s
I20260709 13:01:00.803912  8773 tablet_bootstrap.cc:492] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.804442  8773 ts_tablet_manager.cc:1403] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.042s	user 0.005s	sys 0.007s
I20260709 13:01:00.807927  8773 raft_consensus.cc:359] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.808676  8773 raft_consensus.cc:385] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.809104  8773 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.808421  8775 raft_consensus.cc:359] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.809711  8775 raft_consensus.cc:385] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.810034  8775 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.809842  8773 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.810738  8775 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.812358  8773 ts_tablet_manager.cc:1434] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:01:00.813959  8775 ts_tablet_manager.cc:1434] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 13:01:00.814749  8775 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.815066  8773 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:00.819768  8774 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.820277  8774 ts_tablet_manager.cc:1403] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.082s	user 0.019s	sys 0.010s
I20260709 13:01:00.821863  8775 tablet_bootstrap.cc:654] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.822459  8773 tablet_bootstrap.cc:654] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.823565  8774 raft_consensus.cc:359] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.824267  8774 raft_consensus.cc:385] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.824612  8774 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.825389  8774 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.828660  8774 ts_tablet_manager.cc:1434] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:01:00.829985  8774 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:00.831034  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:00.837383  8926 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.838335  8965 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:00.838142  8926 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.839064  8965 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:00.840205  8965 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.841260  8926 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.838009  8774 tablet_bootstrap.cc:654] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:00.845355  8775 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.845901  8775 ts_tablet_manager.cc:1403] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.031s	user 0.018s	sys 0.011s
I20260709 13:01:00.849305  8775 raft_consensus.cc:359] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.850265  8775 raft_consensus.cc:385] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.850625  8775 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.854724  8654 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.855015  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:00.858023  8775 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.862426  8775 ts_tablet_manager.cc:1434] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.001s
I20260709 13:01:00.862681  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.863870  8775 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.873742  8775 tablet_bootstrap.cc:654] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
W20260709 13:01:00.880198  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 195585us
I20260709 13:01:00.881795  8966 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:00.890129  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.890829  8581 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:00.899533  8775 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:00.900015  8775 ts_tablet_manager.cc:1403] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.009s
I20260709 13:01:00.902679  8775 raft_consensus.cc:359] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.903373  8775 raft_consensus.cc:385] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:00.903656  8775 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:00.905798  8983 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.908470  8983 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.911728  8983 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:00.940883  8775 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.946957  8775 ts_tablet_manager.cc:1434] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.047s	user 0.009s	sys 0.000s
I20260709 13:01:00.948828  8775 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:00.955423  8773 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:00.955952  8773 ts_tablet_manager.cc:1403] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.141s	user 0.021s	sys 0.044s
I20260709 13:01:00.956804  8984 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:00.961277  8658 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:00.965883  8774 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:00.966486  8774 ts_tablet_manager.cc:1403] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.137s	user 0.038s	sys 0.062s
I20260709 13:01:00.966013  8984 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:00.961737  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:00.971779  8549 leader_election.cc:343] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.972806  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:00.973675  8984 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:00.977451  8549 leader_election.cc:343] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:00.990806  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:00.993054  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:00.994426  9002 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:00.994362  8696 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:00.997886  9003 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:00.998459  9003 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:00.998929  9003 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.014357  9003 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.017023  9003 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.018847  9003 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.024062  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:01.024843  8582 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.037196  8582 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:01.039350  8696 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:01.040688  8919 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:01.042815  8919 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:01.043291  8966 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.043819  8919 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.047437  9003 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.047520  8814 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:01.048281  8698 leader_election.cc:343] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.048018  9003 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.048064  8814 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.050168  9003 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.050570  8814 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.051239  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:01:01.053767  8698 consensus_peers.cc:597] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:01:01.054720  8698 leader_election.cc:343] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.054598  8919 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.056111  8919 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.058746  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:01.059134  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:01.061079  8698 leader_election.cc:343] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:01.063903  8698 leader_election.cc:343] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.066651  8773 raft_consensus.cc:359] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.068018  8773 raft_consensus.cc:385] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.068459  8773 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.069356  8773 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.081737  8773 ts_tablet_manager.cc:1434] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.125s	user 0.013s	sys 0.003s
I20260709 13:01:01.082832  8773 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:01.083851  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.090613  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:01.098412  8696 leader_election.cc:343] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.098953  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:01:01.100425  8696 leader_election.cc:343] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.100878  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.101739  8890 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.102353  8890 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.103991  8773 tablet_bootstrap.cc:654] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.111724  8773 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.112208  8773 ts_tablet_manager.cc:1403] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.030s	user 0.009s	sys 0.003s
I20260709 13:01:01.114914  8773 raft_consensus.cc:359] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.115617  8773 raft_consensus.cc:385] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.115880  8773 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.116577  8773 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.119403  8773 ts_tablet_manager.cc:1434] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:01:01.120488  8773 tablet_bootstrap.cc:492] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:01.126494  8773 tablet_bootstrap.cc:654] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.131587  8773 tablet_bootstrap.cc:492] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.132054  8773 ts_tablet_manager.cc:1403] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.012s	user 0.009s	sys 0.001s
I20260709 13:01:01.129944  8774 raft_consensus.cc:359] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.134902  8773 raft_consensus.cc:359] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.135550  8774 raft_consensus.cc:385] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.135696  8773 raft_consensus.cc:385] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.135919  8774 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.136089  8773 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.136660  8774 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.137035  8773 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.139444  8773 ts_tablet_manager.cc:1434] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:01:01.140343  8773 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:01.141148  8775 tablet_bootstrap.cc:654] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.141153  8965 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.144366  8965 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.145917  8890 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.146421  8890 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.147054  8773 tablet_bootstrap.cc:654] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.148310  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:01.150686  8625 leader_election.cc:343] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.151238  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:01.153419  8698 leader_election.cc:343] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.154327  8890 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.157536  8773 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.158002  8773 ts_tablet_manager.cc:1403] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.003s
I20260709 13:01:01.159195  8965 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.160956  8773 raft_consensus.cc:359] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.161753  8773 raft_consensus.cc:385] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.162003  8773 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.162561  8773 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.163961  8773 ts_tablet_manager.cc:1434] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:01:01.162792  8774 ts_tablet_manager.cc:1434] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.196s	user 0.010s	sys 0.001s
I20260709 13:01:01.164646  8773 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:01.166877  8775 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:01.167609  8775 ts_tablet_manager.cc:1403] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.219s	user 0.011s	sys 0.016s
I20260709 13:01:01.170365  8775 raft_consensus.cc:359] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.171051  8773 tablet_bootstrap.cc:654] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.171088  8775 raft_consensus.cc:385] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.171859  8775 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.172959  8775 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.176415  8775 ts_tablet_manager.cc:1434] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.004s
I20260709 13:01:01.177716  8775 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:01.188035  8775 tablet_bootstrap.cc:654] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.188491  8774 tablet_bootstrap.cc:492] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:01.192026  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.193226  8733 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:01.192230  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.193989  8732 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:01.202775  8984 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.203557  8984 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.209795  8984 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.210393  8919 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.213819  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.229462  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.236455  8775 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:01.236958  8775 ts_tablet_manager.cc:1403] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.059s	user 0.013s	sys 0.007s
I20260709 13:01:01.241297  8775 raft_consensus.cc:359] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.245532  8775 raft_consensus.cc:385] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.245937  8775 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.241238  8481 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:01.246807  8775 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.250051  8774 tablet_bootstrap.cc:654] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.252265  8890 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:01:01.252465  8696 leader_election.cc:343] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.254314  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:01.256057  8622 leader_election.cc:343] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.256697  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:01:01.259161  8549 leader_election.cc:343] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.260123  8926 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.260200  8965 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.253126  8696 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.262229  8965 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.263510  8890 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.266567  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.267437  8732 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:01.266878  8965 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.260977  8775 ts_tablet_manager.cc:1434] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.002s
I20260709 13:01:01.270862  8775 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:01.273370  8550 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.278448  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.279589  8984 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:01.279428  8965 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.294533  8984 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.294661  8965 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.279388  8859 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:01.295744  8859 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.295074  8984 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.296396  8859 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.304924  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.307979  8582 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:01.309124  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:01.310151  8926 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:01.310549  8926 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.310870  8926 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.316536  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.317948  8732 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:01.313879  8965 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.323228  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.327289  8859 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.327349  8926 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.330013  8926 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.330765  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:01.331069  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:01.331507  8984 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.331395  8583 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.335742  8859 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.336158  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:01.336742  8733 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.339041  8773 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.339574  8773 ts_tablet_manager.cc:1403] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.175s	user 0.046s	sys 0.082s
W20260709 13:01:01.340447  8549 leader_election.cc:343] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.341547  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:01.344501  8773 raft_consensus.cc:359] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.344832  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.345285  8773 raft_consensus.cc:385] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:01:01.346071  8549 leader_election.cc:343] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.345956  8773 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.348966  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:01.353536  8998 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.353720  8773 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.356005  8966 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.355221  8984 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.341526  8926 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.357537  8926 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.360426  8775 tablet_bootstrap.cc:654] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.345747  8733 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:01.362498  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.363166  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:01.347951  8583 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:01.362572  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.364955  8773 ts_tablet_manager.cc:1434] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 13:01:01.367265  9001 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:01.369002  9001 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:01:01.369181  8625 leader_election.cc:343] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.360459  8926 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.375530  8773 tablet_bootstrap.cc:492] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
W20260709 13:01:01.378211  8622 leader_election.cc:343] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.378759  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:01.378530  9001 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:01.372349  8550 leader_election.cc:343] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.384267  8550 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.388319  8966 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.388880  8859 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.380816  9003 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.390568  9003 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.392518  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:01.393093  8732 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.391669  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:01.393654  8733 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.403774  8859 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.385197  8926 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.406661  8966 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.409785  8966 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.411784  8966 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.412608  8732 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:01.414199  9003 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.415264  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:01.416676  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.417902  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.418689  8584 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:01.419492  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:01:01.421370  8698 leader_election.cc:343] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.422996  8859 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:01:01.422988  8696 leader_election.cc:343] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.424275  8696 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.426765  8773 tablet_bootstrap.cc:654] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.429268  8859 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:01.430055  8652 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.432818  8774 tablet_bootstrap.cc:492] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:01.433318  8774 ts_tablet_manager.cc:1403] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.245s	user 0.007s	sys 0.013s
I20260709 13:01:01.441030  8775 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:01.441581  8775 ts_tablet_manager.cc:1403] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.171s	user 0.017s	sys 0.015s
I20260709 13:01:01.445715  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.446604  8732 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:01.447772  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:01.430603  8859 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.449604  8926 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:01.450842  8965 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.451395  8775 raft_consensus.cc:359] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.452152  8775 raft_consensus.cc:385] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.452381  8733 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:01.452505  8775 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.453380  8775 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.454115  8550 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.454960  8984 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:01.455468  8984 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:01.457850  8984 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.460732  8965 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.456990  8775 ts_tablet_manager.cc:1434] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 13:01:01.467926  8482 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:01.432252  9003 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.472141  8775 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:01.477881  8774 raft_consensus.cc:359] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.480646  8774 raft_consensus.cc:385] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.481045  8774 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.481876  8774 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:01.535418  9025 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.062s	user 0.002s	sys 0.000s
W20260709 13:01:01.538770  8622 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:01.539623  8773 tablet_bootstrap.cc:492] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.540149  8773 ts_tablet_manager.cc:1403] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.165s	user 0.022s	sys 0.006s
I20260709 13:01:01.543829  8481 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:01.546263  8774 ts_tablet_manager.cc:1434] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.113s	user 0.015s	sys 0.010s
I20260709 13:01:01.548241  8480 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:01.552294  8774 tablet_bootstrap.cc:492] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:01.562906  8775 tablet_bootstrap.cc:654] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.568200  8658 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:01.571801  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:01:01.605464  8773 raft_consensus.cc:359] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.606418  8773 raft_consensus.cc:385] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.606827  8773 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.608260  8773 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.612177  8890 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:01.614392  8774 tablet_bootstrap.cc:654] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.615154  9001 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.615761  9001 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.610061  8773 ts_tablet_manager.cc:1434] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.070s	user 0.010s	sys 0.003s
I20260709 13:01:01.618151  8773 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:01.626147  9032 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.628940  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.629236  9032 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.631392  9032 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.626713  9001 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.625348  8773 tablet_bootstrap.cc:654] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.633245  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:01.643538  8773 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.644094  8773 ts_tablet_manager.cc:1403] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.000s
I20260709 13:01:01.647046  8773 raft_consensus.cc:359] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.647799  8773 raft_consensus.cc:385] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.648087  8773 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.648811  8890 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.649309  8890 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.653002  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:01.654412  8698 leader_election.cc:343] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.654435  8919 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:01.655118  8890 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.648794  8773 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.660018  8773 ts_tablet_manager.cc:1434] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:01:01.661624  8773 tablet_bootstrap.cc:492] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:01.663547  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:01.668073  8773 tablet_bootstrap.cc:654] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.668964  8774 tablet_bootstrap.cc:492] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:01.669514  8774 ts_tablet_manager.cc:1403] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.118s	user 0.014s	sys 0.010s
I20260709 13:01:01.673607  8919 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.674332  8919 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.676846  8774 raft_consensus.cc:359] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.677670  8774 raft_consensus.cc:385] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.677989  8774 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.678699  8774 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.679559  8773 tablet_bootstrap.cc:492] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.680032  8773 ts_tablet_manager.cc:1403] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.008s
I20260709 13:01:01.682485  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:01.682607  8773 raft_consensus.cc:359] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.683259  8773 raft_consensus.cc:385] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.683554  8773 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
W20260709 13:01:01.683863  8698 leader_election.cc:343] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.684801  8919 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.684273  8773 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.687219  8773 ts_tablet_manager.cc:1434] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:01:01.687326  8774 ts_tablet_manager.cc:1434] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 13:01:01.688261  8774 tablet_bootstrap.cc:492] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:01.688261  8773 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:01.692713  9032 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.693264  9032 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.694715  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.695956  8984 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.698529  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.694725  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.699928  8481 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:01.705930  8622 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:01.705606  8481 catalog_manager.cc:5697] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:01.706681  9032 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:01.707293  9001 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:01.696203  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.709803  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:01.711468  8550 leader_election.cc:343] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.713981  8919 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.715261  8919 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.720940  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:01.721879  8652 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:01.722433  8919 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:01:01.713503  8696 leader_election.cc:343] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.723212  8696 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.723506  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:01.725226  8652 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:01.725680  8775 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:01.726133  8775 ts_tablet_manager.cc:1403] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.254s	user 0.016s	sys 0.000s
W20260709 13:01:01.726225  8696 leader_election.cc:343] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.726320  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:01.727574  8919 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.727807  8814 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:01.728264  8814 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.728632  8814 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.729374  8775 raft_consensus.cc:359] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.730089  8775 raft_consensus.cc:385] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:01.730379  8775 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:01.731019  8775 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.726613  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:01.740633  9032 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:01.742597  9032 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:01.743083  9032 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.737975  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.748989  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.752132  8877 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.754608  8814 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:01.756268  8696 leader_election.cc:343] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.757663  8775 ts_tablet_manager.cc:1434] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 13:01:01.762171  8775 tablet_bootstrap.cc:492] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:01.771916  8814 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.775702  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:01.777064  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:01.777334  8774 tablet_bootstrap.cc:654] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.777710  8654 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.782101  8814 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.782557  8814 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.781847  9032 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.784024  8654 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:01.784650  8814 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:01.785058  9032 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:01.786062  8696 leader_election.cc:343] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.786506  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:01.787091  8658 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.788738  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:01.789099  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:01:01.790619  8550 leader_election.cc:343] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:01.792208  8698 leader_election.cc:343] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.794263  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:01.795156  8890 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:01.795507  8775 tablet_bootstrap.cc:654] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.795785  8890 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:01.796491  8658 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:01.796545  8890 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.800626  8773 tablet_bootstrap.cc:654] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:01.843464  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:01.862847  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.882553  8773 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:01.886096  8984 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:01:01.886224  8696 leader_election.cc:343] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:01.887349  8696 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:01.895474  9003 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:01.896893  8775 tablet_bootstrap.cc:492] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:01.898109  8775 ts_tablet_manager.cc:1403] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.136s	user 0.012s	sys 0.003s
I20260709 13:01:01.898773  8774 tablet_bootstrap.cc:492] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:01.943552  9001 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.944617  9001 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.962654  8774 ts_tablet_manager.cc:1403] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.275s	user 0.014s	sys 0.007s
I20260709 13:01:01.965497  8773 ts_tablet_manager.cc:1403] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.277s	user 0.012s	sys 0.016s
I20260709 13:01:01.971990  8926 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:01:01.976989  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08001fd2e0 after lost signal to thread 7484
W20260709 13:01:01.979290  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08000dda20 after lost signal to thread 8516
I20260709 13:01:01.982888  8652 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.985570  8926 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:01.989117  9032 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:01.989887  8926 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:01.984524  8984 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:01.994521  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:01.995467  8582 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:01.996793  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:01.981324  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08000d16a0 after lost signal to thread 8608
I20260709 13:01:01.998368  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:01:01.999488  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800256d60 after lost signal to thread 8683
I20260709 13:01:01.999712  8965 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.000447  8965 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.000912  8965 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:01.995517  8984 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.006685  9032 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:02.014631  8775 raft_consensus.cc:359] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.015257  9001 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.015575  8775 raft_consensus.cc:385] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.015988  8775 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.019481  8582 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.016806  8775 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:02.038643  8622 leader_election.cc:343] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:02.009416  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6f20 after lost signal to thread 8757
I20260709 13:01:02.044103  8926 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.067530  8775 ts_tablet_manager.cc:1434] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.169s	user 0.008s	sys 0.004s
I20260709 13:01:02.054128  8965 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.071439  9001 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.077376  8775 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:02.080796  8965 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.081979  9001 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.086768  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:02.087556  8584 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.096019  9047 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:02.098119  8774 raft_consensus.cc:359] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.099128  8774 raft_consensus.cc:385] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.099500  8774 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.100513  8774 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.099499  8773 raft_consensus.cc:359] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.105232  8773 raft_consensus.cc:385] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.105890  8773 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.120149  9001 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.117367  8773 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.120293  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:02.126439  8480 catalog_manager.cc:5697] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:02.123427  8479 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:02.148471  8774 ts_tablet_manager.cc:1434] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.185s	user 0.009s	sys 0.002s
I20260709 13:01:02.150615  8774 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:02.168224  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.169176  8732 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:02.153635  8775 tablet_bootstrap.cc:654] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.171540  8773 ts_tablet_manager.cc:1434] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.185s	user 0.009s	sys 0.002s
I20260709 13:01:02.176239  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.176434  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:02.182760  8584 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:02.180516  9047 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:01:02.189116  9048 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.189761  9048 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.189781  8773 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:02.203773  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.217868  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:02.224198  8926 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.224884  8926 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.225327  8926 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.228788  8773 tablet_bootstrap.cc:654] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.246605  8877 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:01:02.233702  8625 leader_election.cc:343] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:02.252965  8625 leader_election.cc:343] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.256699  9048 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.258342  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:02.259312  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:02.259765  8733 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:02.256050  8775 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:02.264016  8775 ts_tablet_manager.cc:1403] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.187s	user 0.013s	sys 0.007s
I20260709 13:01:02.264984  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:02.266316  9048 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.266927  9048 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.267424  9048 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.266800  8625 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:02.285147  8965 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:02.288022  8926 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:02.297351  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44421.4us
I20260709 13:01:02.290518  8926 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.313634  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:02.313778  9048 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.315505  8579 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:02.310320  8775 raft_consensus.cc:359] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.318113  8775 raft_consensus.cc:385] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.318440  8775 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.320276  8775 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.321542  8965 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:02.322824  8965 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:02.323719  8965 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.325115  8774 tablet_bootstrap.cc:654] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.327163  8775 ts_tablet_manager.cc:1434] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.063s	user 0.006s	sys 0.001s
I20260709 13:01:02.328656  8775 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:02.330360  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:02.332660  8773 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:02.333317  8773 ts_tablet_manager.cc:1403] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.144s	user 0.017s	sys 0.020s
I20260709 13:01:02.339133  9050 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:02.347571  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:02.348343  8733 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.348450  9048 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.388761  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:02.386242  8733 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:02.392916  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.392935  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:02.395414  9032 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:02.395972  9032 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:02.396914  9032 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.394652  8658 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:01:02.401391  8774 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:02.401926  8774 ts_tablet_manager.cc:1403] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.252s	user 0.014s	sys 0.021s
I20260709 13:01:02.403970  8775 tablet_bootstrap.cc:654] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.405864  8774 raft_consensus.cc:359] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.406761  8774 raft_consensus.cc:385] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.407063  8774 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.407817  8774 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.412559  8773 raft_consensus.cc:359] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.413457  8773 raft_consensus.cc:385] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.413746  8773 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.429225  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:02.430168  8658 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:01:02.432549  8774 ts_tablet_manager.cc:1434] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 13:01:02.419691  8773 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.439508  8481 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:02.446081  8774 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:02.444190  8479 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:02.444190  8773 ts_tablet_manager.cc:1434] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.110s	user 0.007s	sys 0.000s
I20260709 13:01:02.461841  9003 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.462023  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:02.462435  9003 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.463120  8926 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:01:02.465734  8775 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:02.465708  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:02.466043  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.466439  8775 ts_tablet_manager.cc:1403] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.138s	user 0.015s	sys 0.005s
I20260709 13:01:02.466912  8658 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:02.468454  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:02.469198  9003 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:02.469597  8877 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.469432  8775 raft_consensus.cc:359] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.470165  8775 raft_consensus.cc:385] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.470470  8775 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.471233  8775 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.478979  8773 tablet_bootstrap.cc:492] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:02.482939  8775 ts_tablet_manager.cc:1434] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:01:02.498101  8877 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:02.499869  8774 tablet_bootstrap.cc:654] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.502822  8926 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.503409  8926 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.505542  8926 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.506608  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:02.513441  8622 leader_election.cc:343] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.519491  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.520255  8730 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:02.521763  8775 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:02.522593  9047 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:02.528033  8625 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:02.529313  8926 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.529856  8926 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.530292  8926 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.537596  8926 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.538305  8773 tablet_bootstrap.cc:654] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.540967  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:01:02.544416  8622 leader_election.cc:343] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.544548  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:02.545261  8729 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.546396  8877 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.549886  8877 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.541882  8926 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.556285  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:02.558772  8696 leader_election.cc:343] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.562286  8877 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:02.566133  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.567036  8774 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:02.566843  9032 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:02.567497  8774 ts_tablet_manager.cc:1403] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.122s	user 0.005s	sys 0.007s
W20260709 13:01:02.568359  8698 leader_election.cc:343] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.569020  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:02.572151  8919 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:02.573190  8774 raft_consensus.cc:359] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.573976  8774 raft_consensus.cc:385] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.574986  8774 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.582996  8775 tablet_bootstrap.cc:654] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.585512  8774 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.603642  8774 ts_tablet_manager.cc:1434] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.000s
I20260709 13:01:02.604878  8774 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:02.607049  9032 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.607630  9032 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.610230  9032 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.611454  8773 tablet_bootstrap.cc:492] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:02.611984  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.607062  8729 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:02.612658  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.616511  8625 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:02.618599  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:02.612022  8773 ts_tablet_manager.cc:1403] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.133s	user 0.032s	sys 0.023s
I20260709 13:01:02.624821  8774 tablet_bootstrap.cc:654] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.627781  9048 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:02.629796  8965 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:02.634440  8965 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:02.635317  8965 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.635486  8773 raft_consensus.cc:359] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.636297  8773 raft_consensus.cc:385] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.636592  8773 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.654027  8775 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:02.654619  8775 ts_tablet_manager.cc:1403] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.133s	user 0.018s	sys 0.017s
I20260709 13:01:02.655014  8774 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:02.655692  8773 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.658834  8773 ts_tablet_manager.cc:1434] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.001s
I20260709 13:01:02.660158  8773 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:02.656185  8774 ts_tablet_manager.cc:1403] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.051s	user 0.004s	sys 0.006s
I20260709 13:01:02.665992  8774 raft_consensus.cc:359] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.666675  8774 raft_consensus.cc:385] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.666952  8774 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.667555  8774 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.674050  8584 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.674881  8773 tablet_bootstrap.cc:654] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.657861  8775 raft_consensus.cc:359] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.683568  8775 raft_consensus.cc:385] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.683956  8775 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.685056  9050 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.685601  9050 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.684795  8775 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.692688  9050 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.698580  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:02.700702  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.701539  8732 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:02.701762  8774 ts_tablet_manager.cc:1434] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.041s	user 0.006s	sys 0.000s
I20260709 13:01:02.706581  8625 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:02.708222  9050 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.708664  9050 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.709034  9050 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:02.711969  8622 leader_election.cc:343] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.713611  8775 ts_tablet_manager.cc:1434] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.059s	user 0.007s	sys 0.004s
I20260709 13:01:02.717823  8774 tablet_bootstrap.cc:492] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:02.720196  8773 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:02.720714  8773 ts_tablet_manager.cc:1403] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.061s	user 0.021s	sys 0.010s
I20260709 13:01:02.722251  8775 tablet_bootstrap.cc:492] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:02.723016  8919 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.723529  8919 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.724328  8773 raft_consensus.cc:359] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.725005  8773 raft_consensus.cc:385] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:01:02.726100  8625 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:02.725258  8773 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.734163  8480 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:02.735319  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:02.736037  8579 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:02.737658  8696 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:02.737573  8773 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.738546  8877 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.741781  8877 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.742163  8877 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.740192  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.740715  8919 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:01:02.747027  8698 leader_election.cc:343] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.748973  9050 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.751401  9050 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.752038  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:01:02.762539  8622 leader_election.cc:343] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.758512  8774 tablet_bootstrap.cc:654] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.763371  8877 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.759186  8926 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:01:02.769037  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:01:02.753623  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:02.769801  8732 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.770363  8775 tablet_bootstrap.cc:654] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.770526  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:02.771179  8579 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.788244  8579 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:02.797760  8696 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:02.798061  8774 tablet_bootstrap.cc:492] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:02.798626  8774 ts_tablet_manager.cc:1403] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.096s	user 0.016s	sys 0.001s
I20260709 13:01:02.799451  8984 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.800036  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:02.799938  8984 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:02.801427  8698 leader_election.cc:343] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.801942  8877 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:02.801628  8774 raft_consensus.cc:359] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.802434  8774 raft_consensus.cc:385] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.802707  8877 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:02.802765  8774 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.798352  8773 ts_tablet_manager.cc:1434] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.077s	user 0.006s	sys 0.001s
I20260709 13:01:02.803716  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.803208  8877 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:02.804975  8877 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.803759  8774 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.805354  8733 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:02.806473  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.802552  8984 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:02.808240  8549 leader_election.cc:343] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.810640  8926 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.810684  8550 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:02.811275  8926 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.812005  8984 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.812397  8984 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.813897  8773 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:02.815004  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:02.815809  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.816064  8774 ts_tablet_manager.cc:1434] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 13:01:02.814504  8926 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.816399  8733 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:02.812731  8984 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.817174  8774 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Bootstrap starting.
W20260709 13:01:02.825724  8622 leader_election.cc:343] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.830163  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:02.831110  8926 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:02.831503  8926 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:02.831861  8926 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.844702  8732 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:02.850446  8773 tablet_bootstrap.cc:654] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.854297  8775 tablet_bootstrap.cc:492] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:02.854774  8775 ts_tablet_manager.cc:1403] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.133s	user 0.021s	sys 0.008s
I20260709 13:01:02.860590  8625 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:02.868103  9050 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:02.868628  9050 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:02.869349  8775 raft_consensus.cc:359] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.872002  8774 tablet_bootstrap.cc:654] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:02.874250  8775 raft_consensus.cc:385] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:02.874615  8775 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:02.872134  8984 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.875409  8775 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.872991  9034 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:02.887151  9034 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.887624  9047 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:02.887540  8926 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.908780  9034 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.926725  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.928046  8984 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.933414  8775 ts_tablet_manager.cc:1434] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.078s	user 0.009s	sys 0.003s
I20260709 13:01:02.933246  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:02.934027  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:02.869477  9050 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:02.934863  8622 leader_election.cc:343] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.930222  9003 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:02.938364  8549 leader_election.cc:343] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:02.939218  8549 leader_election.cc:343] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.948653  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:02.949278  8733 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.950783  9003 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:02.953599  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:02.954607  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:02.955336  8730 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:02.944885  8926 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:02.956477  8481 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:02.960601  8775 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:02.964555  8733 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:02.968575  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:02.953632  9003 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:02.975075  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:01:02.980607  8550 leader_election.cc:343] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:02.981176  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:02.982045  9048 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:02.984591  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:02.971141  9047 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 13:01:02.979323  8965 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:02.996178  8773 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:02.996745  8773 ts_tablet_manager.cc:1403] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.183s	user 0.009s	sys 0.013s
W20260709 13:01:02.999894  8696 leader_election.cc:343] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.000993  8965 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:03.003086  8774 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.004134  8965 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.010588  8774 ts_tablet_manager.cc:1403] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.194s	user 0.014s	sys 0.005s
I20260709 13:01:03.014598  8730 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
W20260709 13:01:03.015910  8698 leader_election.cc:343] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.016376  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.019246  9003 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.032492  8774 raft_consensus.cc:359] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.040017  8775 tablet_bootstrap.cc:654] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.037124  8773 raft_consensus.cc:359] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.041267  8774 raft_consensus.cc:385] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.041603  8773 raft_consensus.cc:385] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.041790  8774 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.041955  8773 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.042902  8774 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.051000  8773 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.062507  8550 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.064465  9032 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:03.065073  9032 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:03.066566  8774 ts_tablet_manager.cc:1434] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.055s	user 0.006s	sys 0.001s
I20260709 13:01:03.066238  9032 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.067580  8774 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.070021  8773 ts_tablet_manager.cc:1434] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.073s	user 0.007s	sys 0.002s
I20260709 13:01:03.064778  8481 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:03.082124  8773 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:03.096596  8774 tablet_bootstrap.cc:654] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
W20260709 13:01:03.104679  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:03.107601  8773 tablet_bootstrap.cc:654] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.111306  8775 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.111869  8775 ts_tablet_manager.cc:1403] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.163s	user 0.004s	sys 0.020s
I20260709 13:01:03.116159  8965 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.116853  8965 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.120221  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.121023  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.118886  8479 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:03.122578  8732 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
W20260709 13:01:03.121841  8622 leader_election.cc:343] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.129050  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:03.130868  9050 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:03.131259  9050 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.131575  9050 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.132367  8774 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.132824  8774 ts_tablet_manager.cc:1403] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.065s	user 0.016s	sys 0.011s
I20260709 13:01:03.135818  8774 raft_consensus.cc:359] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.136502  8774 raft_consensus.cc:385] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.136773  8774 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.137439  8774 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.138609  8965 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.139221  8774 ts_tablet_manager.cc:1434] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:01:03.140280  8774 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.140434  9050 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.143261  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:03.143808  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:03.144367  8732 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.146111  8774 tablet_bootstrap.cc:654] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.146162  8481 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:03.149853  8622 leader_election.cc:343] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.162792  9050 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.161710  8775 raft_consensus.cc:359] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.166488  8775 raft_consensus.cc:385] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.166821  8775 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.169183  8814 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.169929  8774 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.169682  8814 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.170401  8774 ts_tablet_manager.cc:1403] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.004s
I20260709 13:01:03.172703  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.173087  8774 raft_consensus.cc:359] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.173468  8732 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:03.173828  8774 raft_consensus.cc:385] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.174108  8774 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
W20260709 13:01:03.174289  8696 leader_election.cc:343] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.174723  8774 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.175993  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:03.176934  8774 ts_tablet_manager.cc:1434] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:01:03.176857  9082 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:03.177420  9082 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:03.177899  8774 tablet_bootstrap.cc:492] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.178107  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.178227  9082 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:03.179587  8698 leader_election.cc:343] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.179994  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.181169  9086 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.167698  8775 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.183614  8775 ts_tablet_manager.cc:1434] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.071s	user 0.007s	sys 0.002s
I20260709 13:01:03.185680  9003 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.186288  9003 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.188465  9003 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:03.189155  8774 tablet_bootstrap.cc:654] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.190595  8773 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:03.191103  8773 ts_tablet_manager.cc:1403] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.110s	user 0.009s	sys 0.004s
I20260709 13:01:03.185763  8775 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:03.193022  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.197800  8775 tablet_bootstrap.cc:654] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.200698  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.204193  9080 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:03.205744  8696 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.206787  9003 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.212889  9086 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:03.218024  8814 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:03.224015  8774 tablet_bootstrap.cc:492] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.224637  8774 ts_tablet_manager.cc:1403] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.006s
I20260709 13:01:03.228309  8481 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:03.235392  8774 raft_consensus.cc:359] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.235877  8774 raft_consensus.cc:385] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.236035  8774 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.236562  8774 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.234751  8773 raft_consensus.cc:359] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.237332  8773 raft_consensus.cc:385] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.237638  8773 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.238682  8774 ts_tablet_manager.cc:1434] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 13:01:03.238581  8773 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.240250  8774 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.241020  8773 ts_tablet_manager.cc:1434] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.000s
I20260709 13:01:03.242682  8775 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.243146  8775 ts_tablet_manager.cc:1403] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.058s	user 0.011s	sys 0.015s
I20260709 13:01:03.248826  8773 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:03.276252  9032 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.276794  9032 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.279559  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.280108  9032 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.331259  8773 tablet_bootstrap.cc:654] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.332978  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.333773  8732 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:01:03.334968  8550 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.338929  8775 raft_consensus.cc:359] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.339622  8775 raft_consensus.cc:385] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.339902  8775 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.340518  8775 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.343431  9034 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.344689  8775 ts_tablet_manager.cc:1434] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.101s	user 0.019s	sys 0.026s
I20260709 13:01:03.345839  8775 tablet_bootstrap.cc:492] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:03.350553  9050 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:03.353063  9034 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.356426  9050 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.356897  9050 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.360376  9050 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:03.353770  8698 consensus_peers.cc:597] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:03.354241  9080 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.354746  8775 tablet_bootstrap.cc:654] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.365042  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.370191  8732 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:03.365106  9080 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.373468  9080 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.369056  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.377064  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:03.378298  8622 leader_election.cc:343] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.380930  8625 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:03.382727  9080 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.383579  8773 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:03.384050  8773 ts_tablet_manager.cc:1403] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.142s	user 0.017s	sys 0.002s
I20260709 13:01:03.385046  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.386742  8732 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:03.387804  8773 raft_consensus.cc:359] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.388489  8773 raft_consensus.cc:385] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.388779  8773 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.388584  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:03.390352  9050 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:03.390728  9050 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.391073  9050 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.389457  8773 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.402235  8773 ts_tablet_manager.cc:1434] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 13:01:03.407399  8774 tablet_bootstrap.cc:700] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Time spent opening tablet: real 0.125s	user 0.015s	sys 0.043s
I20260709 13:01:03.381069  8775 tablet_bootstrap.cc:492] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.408187  8775 ts_tablet_manager.cc:1403] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.063s	user 0.017s	sys 0.001s
I20260709 13:01:03.408215  8774 tablet_bootstrap.cc:654] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.408424  8773 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:03.397938  9050 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.411105  9080 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.419010  9050 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.420575  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:03.421005  9032 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.424248  8732 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.426239  9032 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.428901  8775 raft_consensus.cc:359] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.429637  8775 raft_consensus.cc:385] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.429896  8775 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.430639  8775 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.432720  8775 ts_tablet_manager.cc:1434] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.023s	user 0.002s	sys 0.002s
I20260709 13:01:03.433777  8775 tablet_bootstrap.cc:492] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:03.433997  9034 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.434489  9034 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.435307  9003 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.435755  9003 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.437237  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:03.438710  9003 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:03.441607  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.441609  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.442183  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:03.443576  8698 leader_election.cc:343] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.443713  9032 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.445866  9034 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.446941  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.447445  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.447902  8729 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:01:03.449786  8696 leader_election.cc:343] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.450227  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:01:03.451402  8549 leader_election.cc:343] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.451431  8814 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:01:03.448787  8622 leader_election.cc:343] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.453287  8732 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:03.454777  8550 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.455585  9048 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.457620  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:03.458307  8773 tablet_bootstrap.cc:654] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.462208  9050 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:03.462625  9080 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:03.463788  9050 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:03.462621  8774 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.464752  8774 ts_tablet_manager.cc:1403] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.225s	user 0.021s	sys 0.058s
I20260709 13:01:03.468992  9048 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.464926  9050 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.470680  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:03.472432  8550 leader_election.cc:343] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.472920  8550 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.474321  8774 raft_consensus.cc:359] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.475236  8774 raft_consensus.cc:385] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.475603  8774 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.475766  9048 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.484573  9080 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:03.486377  8579 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.497253  8481 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:03.502305  8774 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.513322  8774 ts_tablet_manager.cc:1434] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.048s	user 0.007s	sys 0.000s
I20260709 13:01:03.514518  8775 tablet_bootstrap.cc:654] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.514518  8774 tablet_bootstrap.cc:492] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.517202  9050 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:03.536734  9032 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:03.555456  8773 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:03.555960  8773 ts_tablet_manager.cc:1403] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.148s	user 0.022s	sys 0.024s
I20260709 13:01:03.558554  8774 tablet_bootstrap.cc:654] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.565143  8773 raft_consensus.cc:359] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.568765  8773 raft_consensus.cc:385] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.569166  8773 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.571357  8773 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.589013  8773 ts_tablet_manager.cc:1434] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.002s
I20260709 13:01:03.595880  9032 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:03.599025  8773 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:03.600841  8775 tablet_bootstrap.cc:492] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.601807  8775 ts_tablet_manager.cc:1403] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.168s	user 0.010s	sys 0.009s
I20260709 13:01:03.605881  8775 raft_consensus.cc:359] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.609345  8775 raft_consensus.cc:385] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.609674  8775 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.618999  8775 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.622079  8775 ts_tablet_manager.cc:1434] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 13:01:03.623255  8775 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:03.634296  9050 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.634810  9050 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.635986  8773 tablet_bootstrap.cc:654] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.636936  9050 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.638890  8774 tablet_bootstrap.cc:492] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.639515  8774 ts_tablet_manager.cc:1403] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.125s	user 0.014s	sys 0.002s
I20260709 13:01:03.643039  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.643391  8774 raft_consensus.cc:359] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.644462  8774 raft_consensus.cc:385] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.644992  8774 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.646149  8877 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.646425  8775 tablet_bootstrap.cc:654] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.646715  8877 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.649972  8774 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.651026  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.652678  8774 ts_tablet_manager.cc:1434] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:01:03.654510  8774 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.659834  8877 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:03.661245  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.662484  8775 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.662575  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.663129  8775 ts_tablet_manager.cc:1403] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.040s	user 0.010s	sys 0.002s
W20260709 13:01:03.662994  8696 leader_election.cc:343] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:03.666389  8698 leader_election.cc:343] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.666211  8775 raft_consensus.cc:359] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.666870  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.667163  8775 raft_consensus.cc:385] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.667512  8775 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.668272  8775 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.669770  8814 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.671522  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:03.673002  9080 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.676882  8775 ts_tablet_manager.cc:1434] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:01:03.678062  8775 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:03.679656  9080 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:03.687012  8774 tablet_bootstrap.cc:654] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.690378  8775 tablet_bootstrap.cc:654] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.691250  8579 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.711711  9050 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:03.713511  8877 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.714008  8877 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.715801  8877 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:03.717667  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.725606  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:03.727819  8698 leader_election.cc:343] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:03.731904  8696 leader_election.cc:343] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.732357  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.733340  8877 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.743747  8730 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.751117  8773 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:03.753337  8773 ts_tablet_manager.cc:1403] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.155s	user 0.017s	sys 0.021s
I20260709 13:01:03.757012  8773 raft_consensus.cc:359] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.761569  8774 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.762476  8774 ts_tablet_manager.cc:1403] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.108s	user 0.028s	sys 0.022s
I20260709 13:01:03.758069  8773 raft_consensus.cc:385] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.764117  8773 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.764842  8773 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.765174  8774 raft_consensus.cc:359] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.765832  8774 raft_consensus.cc:385] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.766106  8774 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.766891  8774 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.768959  8775 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.769083  8774 ts_tablet_manager.cc:1434] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 13:01:03.770162  8774 tablet_bootstrap.cc:492] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.777217  8774 tablet_bootstrap.cc:654] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.769507  8775 ts_tablet_manager.cc:1403] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.092s	user 0.013s	sys 0.000s
I20260709 13:01:03.788861  8774 tablet_bootstrap.cc:492] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.789316  8774 ts_tablet_manager.cc:1403] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.007s
I20260709 13:01:03.795276  8774 raft_consensus.cc:359] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.795958  8774 raft_consensus.cc:385] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.796216  8774 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.796813  8774 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.804813  8774 ts_tablet_manager.cc:1434] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.001s
I20260709 13:01:03.805783  8774 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.812263  8774 tablet_bootstrap.cc:654] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.814191  8775 raft_consensus.cc:359] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.814929  8775 raft_consensus.cc:385] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.815826  8773 ts_tablet_manager.cc:1434] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.062s	user 0.007s	sys 0.000s
I20260709 13:01:03.815660  9034 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:03.822117  8774 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.822170  9034 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.822750  8774 ts_tablet_manager.cc:1403] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.002s
I20260709 13:01:03.824801  9086 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.825313  9086 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.826473  9034 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.827011  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.828043  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.828912  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.830497  8773 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:03.830382  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:01:03.830780  8698 leader_election.cc:343] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.831390  9086 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:03.815258  8775 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.832978  8774 raft_consensus.cc:359] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.834923  8774 raft_consensus.cc:385] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.835256  8774 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.835891  8774 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.837831  9047 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:03.838194  8774 ts_tablet_manager.cc:1434] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.002s
I20260709 13:01:03.838354  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.839291  8774 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Bootstrap starting.
W20260709 13:01:03.842806  8696 leader_election.cc:343] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.843305  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.845038  8877 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.849947  8774 tablet_bootstrap.cc:654] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.860967  8774 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.861429  8774 ts_tablet_manager.cc:1403] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.007s
I20260709 13:01:03.865012  8774 raft_consensus.cc:359] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.865724  8774 raft_consensus.cc:385] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.865980  8774 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.866802  8774 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.871310  8652 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.834189  8775 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.874010  8775 ts_tablet_manager.cc:1434] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.092s	user 0.007s	sys 0.000s
I20260709 13:01:03.875200  8775 tablet_bootstrap.cc:492] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:03.877035  9047 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.877592  9047 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.879792  9047 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.881469  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.882445  8775 tablet_bootstrap.cc:654] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
W20260709 13:01:03.882843  8549 leader_election.cc:343] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.868860  8774 ts_tablet_manager.cc:1434] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:01:03.885035  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.885782  8730 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:03.886183  8774 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.890388  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.891810  9047 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:03.892360  9047 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.892889  9047 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.894217  8773 tablet_bootstrap.cc:654] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.896351  8774 tablet_bootstrap.cc:654] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.907258  8774 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.907732  8774 ts_tablet_manager.cc:1403] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.022s	user 0.002s	sys 0.008s
I20260709 13:01:03.910436  8774 raft_consensus.cc:359] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.911094  9086 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.911772  9086 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.913292  8775 tablet_bootstrap.cc:492] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.914093  8775 ts_tablet_manager.cc:1403] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.039s	user 0.010s	sys 0.023s
I20260709 13:01:03.911104  8774 raft_consensus.cc:385] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.915962  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.916626  8582 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:03.916116  8774 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.917264  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:03.917456  8774 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:03.918711  8698 leader_election.cc:343] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.918613  8775 raft_consensus.cc:359] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.919436  8775 raft_consensus.cc:385] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.919773  8775 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.919950  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.916571  8773 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:03.920773  8773 ts_tablet_manager.cc:1403] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.091s	user 0.013s	sys 0.000s
I20260709 13:01:03.920559  8775 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.920796  9003 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:03.921557  9003 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.921895  9003 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.922966  8775 ts_tablet_manager.cc:1434] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:01:03.923604  8774 ts_tablet_manager.cc:1434] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:01:03.923416  8773 raft_consensus.cc:359] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.924156  8775 tablet_bootstrap.cc:492] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:03.924690  8774 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:03.924108  8773 raft_consensus.cc:385] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.925585  8773 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.925808  8877 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.932523  8877 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.932757  8773 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.934959  8877 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:03.935732  8773 ts_tablet_manager.cc:1434] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:01:03.935981  9082 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.936470  9082 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.936887  8773 tablet_bootstrap.cc:492] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:03.937018  8774 tablet_bootstrap.cc:654] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.931879  8775 tablet_bootstrap.cc:654] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.938975  9082 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.942036  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:03.942745  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 13:01:03.944717  8698 leader_election.cc:343] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.947045  8773 tablet_bootstrap.cc:654] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.947434  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:03.948086  8730 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:03.949376  8625 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:03.951107  9082 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:03.951529  9082 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:03.951857  9082 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.961114  8774 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:03.960983  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.961668  8774 ts_tablet_manager.cc:1403] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.037s	user 0.007s	sys 0.004s
I20260709 13:01:03.961833  8580 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:03.965062  8775 tablet_bootstrap.cc:492] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:03.965749  8775 ts_tablet_manager.cc:1403] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.042s	user 0.014s	sys 0.001s
I20260709 13:01:03.968500  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:03.968565  8775 raft_consensus.cc:359] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.969694  8775 raft_consensus.cc:385] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.970160  8775 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.970980  8775 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.969542  8579 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:03.964728  8774 raft_consensus.cc:359] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.973608  8774 raft_consensus.cc:385] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.973922  8774 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.973989  8775 ts_tablet_manager.cc:1434] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:01:03.975200  8775 tablet_bootstrap.cc:492] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
W20260709 13:01:03.976066  8696 leader_election.cc:343] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:03.976497  8696 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:03.977154  8814 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:03.979029  8773 tablet_bootstrap.cc:492] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:03.979612  8773 ts_tablet_manager.cc:1403] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.043s	user 0.019s	sys 0.000s
I20260709 13:01:03.983839  8773 raft_consensus.cc:359] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.989876  8773 raft_consensus.cc:385] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:03.990293  8773 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:03.974557  8774 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.991113  8773 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:03.994033  8773 ts_tablet_manager.cc:1434] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 13:01:03.995662  8773 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:03.997074  8774 ts_tablet_manager.cc:1434] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.002s
I20260709 13:01:03.989486  8775 tablet_bootstrap.cc:654] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:03.997640  9032 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:03.998128  9032 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.000207  9032 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:03.925577  9086 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:04.003300  8774 tablet_bootstrap.cc:492] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:04.005535  9050 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:04.009037  8773 tablet_bootstrap.cc:654] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:04.011930  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.016103  8774 tablet_bootstrap.cc:654] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:04.018031  8775 tablet_bootstrap.cc:492] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:04.018651  8775 ts_tablet_manager.cc:1403] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.044s	user 0.010s	sys 0.007s
I20260709 13:01:04.021349  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.021653  8775 raft_consensus.cc:359] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.022291  8730 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:04.022553  8775 raft_consensus.cc:385] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:04.023072  8775 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:04.023953  8775 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.025488  8774 tablet_bootstrap.cc:492] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:04.025971  8774 ts_tablet_manager.cc:1403] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.000s
I20260709 13:01:04.027352  8775 ts_tablet_manager.cc:1434] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:01:04.030651  8773 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:04.031212  8773 ts_tablet_manager.cc:1403] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.036s	user 0.010s	sys 0.002s
I20260709 13:01:04.033982  8773 raft_consensus.cc:359] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.035643  8773 raft_consensus.cc:385] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:04.036008  8773 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:04.036727  8773 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.035228  8775 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
W20260709 13:01:04.049953  8549 leader_election.cc:343] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.052532  9082 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.045845  9047 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.059105  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.065142  9047 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.065165  9048 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.066256  9048 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.067042  8774 raft_consensus.cc:359] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.067721  8774 raft_consensus.cc:385] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:04.068272  9048 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.069386  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:04.065282  9034 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.069764  9047 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.070262  9047 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.070190  9034 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.070595  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.070783  9047 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.067981  8774 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
W20260709 13:01:04.073249  8549 leader_election.cc:343] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:04.074086  8549 leader_election.cc:343] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.072461  8774 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.077663  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.080034  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:04.080618  8730 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.078183  8732 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:04.082898  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.054381  9032 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.084162  8775 tablet_bootstrap.cc:654] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:04.088404  9032 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.089802  9082 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.090619  9032 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.091167  9032 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.091408  9034 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.074910  9086 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.092272  9086 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.099550  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.103504  8773 ts_tablet_manager.cc:1434] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.072s	user 0.026s	sys 0.015s
I20260709 13:01:04.102463  8654 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:04.106612  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.106798  8549 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:04.107256  8732 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:04.091492  9032 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.054189  9003 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.113459  9003 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:04.114724  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:04.114933  9086 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:04.114975  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:04.115391  8653 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.125226  8731 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.125671  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:04.122136  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.126339  8583 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.126783  8729 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:04.127983  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:04.125584  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:04.110284  9032 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.139227  9135 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.139775  9135 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.140122  9135 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.142499  8773 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:04.124564  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:04.138687  8580 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:04.149310  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:04.150283  8814 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.150643  8814 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.150954  8814 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.141429  9082 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 13:01:04.169046  9034 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.169296  9003 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.169584  9034 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.137298  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.169799  9003 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.169983  9034 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:04.174480  8549 leader_election.cc:343] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.175051  8579 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.183364  8731 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:04.184589  9003 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:04.186311  8774 ts_tablet_manager.cc:1434] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.160s	user 0.017s	sys 0.007s
I20260709 13:01:04.191246  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:04.195854  8698 leader_election.cc:343] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.216055  8774 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:04.227732  8775 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:04.228212  8775 ts_tablet_manager.cc:1403] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.193s	user 0.011s	sys 0.013s
W20260709 13:01:04.229007  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 101115us
I20260709 13:01:04.203397  8730 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:04.227661  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:04.233080  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.278585  9047 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.283850  9082 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.284654  9082 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.293880  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.294044  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:04.294802  8731 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.284121  9047 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.328555  8814 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.331995  8814 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:01:04.326263  9137 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.112s	user 0.001s	sys 0.001s
W20260709 13:01:04.354247  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60456us
I20260709 13:01:04.355175  8625 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:04.358353  8775 raft_consensus.cc:359] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.359352  8775 raft_consensus.cc:385] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:04.343509  9048 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:04.359730  8775 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
W20260709 13:01:04.370337  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70042.3us
W20260709 13:01:04.370829  8696 leader_election.cc:343] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.371425  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.353929  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:04.364162  8775 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:04.374114  8698 leader_election.cc:343] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.360448  9142 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.377839  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:04.390892  8773 tablet_bootstrap.cc:654] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:04.375355  9003 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:04.416806  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:04.422293  8583 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:04.424558  8775 ts_tablet_manager.cc:1434] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.196s	user 0.007s	sys 0.010s
I20260709 13:01:04.416386  9143 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:04.437779  9048 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:04.451506  9048 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.381877  9142 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:04.458801  8549 leader_election.cc:343] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:04.461253  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 84807.2us
I20260709 13:01:04.442989  9086 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.462517  9086 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.442350  9082 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.442108  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:04.472005  8698 leader_election.cc:343] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.456458  9143 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:04.512722  8814 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.513259  8814 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.535960  8775 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:04.537642  8814 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:04.531479  8731 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:04.569442  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.579437  9146 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.563510  8579 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:04.519316  9143 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.581212  9146 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.602949  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.615747  8774 tablet_bootstrap.cc:654] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:04.613912  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:04.613966  9148 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:04.619781  8580 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:04.616068  9034 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.623340  8584 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:04.612694  9032 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.625166  9148 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:04.613508  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:04.627575  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:04.626292  9034 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.616528  9135 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:04.633788  8549 leader_election.cc:343] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.626075  9148 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:04.639608  8625 leader_election.cc:343] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.640523  9146 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.634779  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.642592  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:04.644114  9142 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:04.646849  8549 leader_election.cc:343] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.647686  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:01:04.647934  8549 leader_election.cc:343] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.648324  8650 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.656006  8775 tablet_bootstrap.cc:654] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:04.657953  9032 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:04.648834  8653 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.514s	user 0.011s	sys 0.007s
I20260709 13:01:04.658676  8653 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:04.649183  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.665306  8724 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:04.671460  8773 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:04.671974  8773 ts_tablet_manager.cc:1403] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.534s	user 0.023s	sys 0.037s
I20260709 13:01:04.682163  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:04.698622  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:04.703951  8814 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:04.704468  8814 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:04.706069  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.705210  8814 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.706722  8733 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:04.703662  9143 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.715762  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:04.715941  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:04.716790  8773 raft_consensus.cc:359] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.720191  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.720552  8773 raft_consensus.cc:385] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:04.738122  8774 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:04.742030  8774 ts_tablet_manager.cc:1403] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.533s	user 0.023s	sys 0.008s
I20260709 13:01:04.742731  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.741190  8773 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
W20260709 13:01:04.740692  8622 leader_election.cc:343] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.721632  8731 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.720415  9143 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.743960  8696 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:04.750304  8726 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:01:04.749779  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:04.755405  8622 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:04.758527  9003 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.758952  9003 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.740156  9139 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:04.759294  9003 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.759467  9139 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:04.759894  9139 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:04.760896  8696 leader_election.cc:343] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.720814  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:04.721792  9086 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:04.761778  9143 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.765803  8773 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.771085  9141 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:04.773149  8773 ts_tablet_manager.cc:1434] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.101s	user 0.006s	sys 0.003s
I20260709 13:01:04.774537  8773 tablet_bootstrap.cc:492] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:04.795814  9135 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.800554  8774 raft_consensus.cc:359] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.802322  8774 raft_consensus.cc:385] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:04.802732  8774 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:04.804659  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.774688  8725 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.814569  8774 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:04.824684  8698 leader_election.cc:343] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.775761  8650 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
W20260709 13:01:04.827204  8698 leader_election.cc:343] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.827644  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.828408  8814 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:04.831516  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.836925  8774 ts_tablet_manager.cc:1434] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.094s	user 0.005s	sys 0.003s
I20260709 13:01:04.813450  9048 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 13:01:04.838557  8759 tablet.cc:2406] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:01:04.842653  8731 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:04.857074  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:04.860003  8877 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:01:04.860607  8774 tablet_bootstrap.cc:492] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:04.877327  9154 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:04.878083  9154 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:04.891058  9162 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.891738  9162 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.912433  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:04.888314  9003 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.915977  9003 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:04.880187  9154 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.916992  8775 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:04.917531  8775 ts_tablet_manager.cc:1403] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.392s	user 0.020s	sys 0.021s
I20260709 13:01:04.917585  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:04.917975  9162 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:04.919577  8698 leader_election.cc:343] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.922564  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.925575  8775 raft_consensus.cc:359] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.926359  8775 raft_consensus.cc:385] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:04.926682  8775 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:04.932730  8725 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
W20260709 13:01:04.934082  8550 leader_election.cc:343] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:04.936758  8773 tablet_bootstrap.cc:654] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:04.943377  8775 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.946385  8775 ts_tablet_manager.cc:1434] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 13:01:04.948694  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:04.963506  9142 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:04.945574  9139 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.965188  9142 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.963119  9136 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:04.966527  9136 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:04.968780  9136 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.954033  9143 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:04.974565  9142 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.982401  9139 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:04.976344  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:04.984373  8725 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:04.985618  8775 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:04.989579  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:04.990413  8725 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:04.996734  8725 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:04.973989  8481 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:04.998034  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:05.000789  9154 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:05.001370  9154 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:05.002370  9154 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:05.009198  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:01:05.017618  9143 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:05.010633  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:05.019824  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:05.022048  8728 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:05.034345  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:05.034797  8622 leader_election.cc:343] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.048732  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:01:05.050813  8549 leader_election.cc:343] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.051441  8549 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:05.052232  9135 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:05.053946  8773 tablet_bootstrap.cc:492] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:05.054324  8773 ts_tablet_manager.cc:1403] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.280s	user 0.021s	sys 0.013s
I20260709 13:01:05.056990  8774 tablet_bootstrap.cc:654] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:05.060979  8775 tablet_bootstrap.cc:654] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:05.060971  8773 raft_consensus.cc:359] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.067845  8728 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:05.068463  8773 raft_consensus.cc:385] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:05.071694  8773 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
W20260709 13:01:05.063876  8549 leader_election.cc:343] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.073521  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:05.074414  9143 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:05.074960  9143 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:05.075342  8773 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.075975  9143 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.077266  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:05.077966  8581 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:05.079425  8773 ts_tablet_manager.cc:1434] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.001s
I20260709 13:01:05.094393  8773 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
W20260709 13:01:05.098999  8549 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:05.090611  8622 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:05.099794  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:05.101703  9142 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:05.107369  8622 leader_election.cc:343] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.109025  8775 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:05.109544  8775 ts_tablet_manager.cc:1403] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.124s	user 0.012s	sys 0.005s
I20260709 13:01:05.114679  9143 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:01:05.118775  8581 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:05.126295  8775 raft_consensus.cc:359] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.127148  8775 raft_consensus.cc:385] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:05.127489  8775 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:05.142748  8773 tablet_bootstrap.cc:654] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:05.132301  8877 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 13:01:05.143874  8775 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.147601  8696 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:05.151299  9151 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:05.151845  9151 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:05.152721  9151 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.155378  8814 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.156152  8775 ts_tablet_manager.cc:1434] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.046s	user 0.008s	sys 0.000s
I20260709 13:01:05.155891  8814 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.157989  8814 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:05.170517  8773 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:05.171089  8773 ts_tablet_manager.cc:1403] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.077s	user 0.012s	sys 0.004s
I20260709 13:01:05.170892  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:05.172714  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:05.142138  9142 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:01:05.174602  8698 leader_election.cc:343] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:05.177354  8696 leader_election.cc:343] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.177834  8696 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:05.180132  9003 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:05.199450  8773 raft_consensus.cc:359] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.200246  8773 raft_consensus.cc:385] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:05.200747  8773 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:05.202551  8775 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:05.202708  8774 tablet_bootstrap.cc:492] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:05.203397  8774 ts_tablet_manager.cc:1403] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.348s	user 0.027s	sys 0.018s
I20260709 13:01:05.180184  9143 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 13:01:05.221529  9086 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:05.201545  8773 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.300370  8773 ts_tablet_manager.cc:1434] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.129s	user 0.007s	sys 0.000s
I20260709 13:01:05.301368  8773 tablet_bootstrap.cc:492] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:05.307034  9145 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:05.332463  8774 raft_consensus.cc:359] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.333266  8774 raft_consensus.cc:385] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:05.335718  8774 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:05.370728  9139 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 13:01:05.435604  9143 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.443779  9143 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.464758  8480 catalog_manager.cc:5697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:05.492545  9143 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:05.503760  8774 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.521802  8774 ts_tablet_manager.cc:1434] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.318s	user 0.010s	sys 0.000s
I20260709 13:01:05.524107  8814 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.524621  8814 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.528756  8814 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:05.535771  9145 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:05.536988  9145 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:05.553079  8773 tablet_bootstrap.cc:654] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:05.555017  9143 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.558315  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:05.558074  9143 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.559082  8726 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:05.559350  8775 tablet_bootstrap.cc:654] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:05.560173  9143 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:05.570261  8774 tablet_bootstrap.cc:492] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:05.597728  9148 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:05.598320  9148 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.600764  9148 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:05.576988  8479 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:05.619035  8479 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:05.623533  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:05.625103  8479 catalog_manager.cc:5697] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:05.627480  8479 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:05.635262  8773 tablet_bootstrap.cc:492] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:05.635947  8773 ts_tablet_manager.cc:1403] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.335s	user 0.017s	sys 0.006s
I20260709 13:01:05.635351  8479 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:05.641561  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:05.642561  8584 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:01:05.646541  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 117861us
I20260709 13:01:05.645314  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:05.657140  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:05.658197  8732 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:05.673280  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:05.652166  9136 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260709 13:01:05.658128  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:05.654759  9147 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260709 13:01:05.719775  8654 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:05.736897  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 84294.9us
I20260709 13:01:05.743268  8775 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:05.743834  8775 ts_tablet_manager.cc:1403] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.550s	user 0.018s	sys 0.025s
I20260709 13:01:05.746238  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:01:05.750967  8698 leader_election.cc:343] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.766727  8773 raft_consensus.cc:359] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.790226  8773 raft_consensus.cc:385] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:05.790573  8773 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:05.792975  8774 tablet_bootstrap.cc:654] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:05.793159  8773 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:05.794384  8549 leader_election.cc:343] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.792603  9154 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:05.803067  9154 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:05.803501  9154 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:05.803831  9198 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
W20260709 13:01:05.825748  9200 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 13:01:05.826196  8773 ts_tablet_manager.cc:1434] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.190s	user 0.008s	sys 0.018s
W20260709 13:01:05.844285  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 90911.2us
I20260709 13:01:05.845880  8775 raft_consensus.cc:359] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.847398  8775 raft_consensus.cc:385] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:05.847674  8775 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
W20260709 13:01:05.850455  9194 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 13:01:05.852177  8775 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.852995  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:05.856904  9154 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:05.861244  9154 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:05.836915  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 81209.8us
W20260709 13:01:05.867359  8696 leader_election.cc:343] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:05.867882  8696 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:05.870561  8775 ts_tablet_manager.cc:1434] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.126s	user 0.005s	sys 0.003s
I20260709 13:01:05.870558  8877 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:05.872083  9148 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:05.872500  8775 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Bootstrap starting.
I20260709 13:01:05.897627  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:01:05.899511  8622 consensus_peers.cc:597] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:05.900339  9208 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:05.878826  9086 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260709 13:01:05.928485  8775 tablet_bootstrap.cc:654] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:05.934559  8773 tablet_bootstrap.cc:492] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:05.935317  8480 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:05.956243  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:05.956986  8727 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:05.970080  8814 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:05.983429  8774 tablet_bootstrap.cc:492] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:06.015844  8774 ts_tablet_manager.cc:1403] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.446s	user 0.033s	sys 0.059s
I20260709 13:01:06.028359  8773 tablet_bootstrap.cc:654] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:06.023504  8774 raft_consensus.cc:359] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.036036  8774 raft_consensus.cc:385] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:06.013274  9208 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:06.037108  8774 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:06.043606  8774 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.047816  8774 ts_tablet_manager.cc:1434] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.000s
I20260709 13:01:06.050352  8625 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:06.035302  9034 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260709 13:01:06.083643  8775 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: No bootstrap required, opened a new log
I20260709 13:01:06.084113  8775 ts_tablet_manager.cc:1403] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent bootstrapping tablet: real 0.212s	user 0.010s	sys 0.008s
I20260709 13:01:06.079795  9142 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:01:06.143987  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67959.4us
I20260709 13:01:06.077538  8727 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:06.241384  8774 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:06.248589  9202 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.249691  9202 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.252636  8773 tablet_bootstrap.cc:492] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:06.253140  8773 ts_tablet_manager.cc:1403] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.319s	user 0.030s	sys 0.059s
I20260709 13:01:06.256350  8774 tablet_bootstrap.cc:654] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:06.261049  8773 raft_consensus.cc:359] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.264021  9202 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.402655  8773 raft_consensus.cc:385] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:06.325702  9148 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.403163  8773 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:06.403324  9148 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.403877  8773 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:06.261808  8549 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:01:06.405943  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 144288us
I20260709 13:01:06.406968  9148 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.414173  8773 ts_tablet_manager.cc:1434] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.161s	user 0.007s	sys 0.002s
W20260709 13:01:06.298913  9218 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.265s	user 0.012s	sys 0.004s
I20260709 13:01:06.420787  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:06.428027  8773 tablet_bootstrap.cc:492] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:06.424888  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:01:06.443216  8549 leader_election.cc:343] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:06.441150  8580 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:06.453182  8775 raft_consensus.cc:359] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.454635  8775 raft_consensus.cc:385] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:06.455045  8775 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Initialized, Role: FOLLOWER
I20260709 13:01:06.455978  8775 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.477344  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:06.483742  8775 ts_tablet_manager.cc:1434] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent starting tablet: real 0.399s	user 0.009s	sys 0.000s
I20260709 13:01:06.490340  9207 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.490883  9207 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.473485  9086 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.492667  9086 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.473291  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:06.505580  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 52968.7us
I20260709 13:01:06.527773  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:06.493317  9207 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:06.500792  9086 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:01:06.537822  8549 leader_election.cc:343] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:06.561586  9230 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:01:06.562399  9203 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:06.519542  8724 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:06.568400  8773 tablet_bootstrap.cc:654] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:06.556725  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:06.598941  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 82408.1us
I20260709 13:01:06.586316  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:06.608383  8774 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:06.608975  8774 ts_tablet_manager.cc:1403] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.387s	user 0.008s	sys 0.023s
W20260709 13:01:06.618324  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 111589us
I20260709 13:01:06.568743  9203 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:06.621949  8652 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:06.601598  8657 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 0.
I20260709 13:01:06.641790  9203 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.643743  8774 raft_consensus.cc:359] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.644542  8774 raft_consensus.cc:385] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:06.644820  8774 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:06.645519  8774 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.647962  8774 ts_tablet_manager.cc:1434] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.000s
I20260709 13:01:06.648934  8774 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:06.652390  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:01:06.641745  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58169.7us
I20260709 13:01:06.654702  9139 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:06.663174  9139 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.663601  9139 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:06.672217  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:06.675203  8773 tablet_bootstrap.cc:492] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:06.675695  8773 ts_tablet_manager.cc:1403] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.254s	user 0.019s	sys 0.014s
I20260709 13:01:06.670118  9003 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.678401  9003 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.703933  9148 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.707552  9148 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.710044  9003 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:06.756623  9148 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.756974  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:06.831394  8729 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:06.812736  9135 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.832350  9135 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.834826  8773 raft_consensus.cc:359] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.835767  8773 raft_consensus.cc:385] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:06.836128  8773 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:06.836611  9135 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.842648  8550 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:06.844532  9211 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:06.845055  9211 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.845609  9211 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:06.845865  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:06.847146  8726 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:01:06.850905  9034 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.851490  9034 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.825683  8774 tablet_bootstrap.cc:654] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:06.853672  9034 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.855669  9211 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.867609  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:06.824661  9139 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.868377  8728 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
W20260709 13:01:06.869163  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 197039us
I20260709 13:01:06.873215  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:06.873308  9205 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:06.875066  9205 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.875463  9205 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:06.878798  9241 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:06.879274  9241 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.880702  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:06.880224  8877 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.881292  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:06.881717  8877 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.883576  8877 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:06.880745  9241 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:06.886129  9142 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:06.886665  9142 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.888337  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:06.889286  9142 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.889806  9136 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:06.890204  9136 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.890538  9136 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:06.892930  9241 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.890997  9205 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:06.894325  8698 leader_election.cc:343] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:06.865744  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:06.895720  8698 leader_election.cc:343] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:06.895797  8726 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 0.
I20260709 13:01:06.898329  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:06.899250  9034 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:06.899798  9034 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:06.900429  9034 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:06.894512  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:06.906627  8733 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:06.909027  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:06.909739  8650 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:01:06.915127  8530 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:01:06.920835  8650 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:01:06.907763  9241 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:06.922413  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:06.906351  9136 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.923103  9139 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.866223  8773 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.923336  9241 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:06.923789  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:06.924034  9241 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:06.924481  8726 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:06.924935  9241 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.925531  9136 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.929239  8773 ts_tablet_manager.cc:1434] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.253s	user 0.005s	sys 0.008s
I20260709 13:01:06.930277  8773 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:06.872236  9211 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.930472  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:06.931314  8732 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 2: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:01:06.932809  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:06.934623  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:06.941042  8773 tablet_bootstrap.cc:654] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:06.921223  8479 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:01:06.946663  9034 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.949359  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:06.950024  8650 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 2
I20260709 13:01:06.950451  8774 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:06.951102  8774 ts_tablet_manager.cc:1403] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.302s	user 0.003s	sys 0.009s
I20260709 13:01:06.952360  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:06.953092  8727 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:06.955376  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:06.957430  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:06.958617  8580 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 2: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:06.958853  8650 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:06.960178  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:06.961145  8727 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:06.961581  9241 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:06.962158  9241 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:06.962352  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:06.963165  9211 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:06.962976  9241 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.963861  9211 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:06.965903  9142 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:06.964746  9211 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.967368  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:06.942263  8726 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:06.972218  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:06.972838  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:06.973678  9136 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:06.948252  9205 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:06.948876  8733 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:06.974184  9136 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:06.975236  8550 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:06.975970  9145 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:06.976270  9034 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:06.975556  9136 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.976605  9145 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:06.974208  8774 raft_consensus.cc:359] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.979342  8774 raft_consensus.cc:385] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:06.979698  8774 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:06.981055  8774 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:06.977532  9145 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.003624  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:06.987730  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:07.006484  8579 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 2: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:06.988873  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:07.011818  8774 ts_tablet_manager.cc:1434] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.060s	user 0.008s	sys 0.001s
I20260709 13:01:07.012552  9139 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:07.025467  8773 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:07.026041  8658 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.026118  8773 ts_tablet_manager.cc:1403] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.096s	user 0.017s	sys 0.012s
I20260709 13:01:07.027172  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:07.032599  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:07.036835  8774 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:07.038187  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:07.040838  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:07.042001  8651 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 2: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:01:07.048527  8696 leader_election.cc:343] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.051075  8773 raft_consensus.cc:359] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.051875  8773 raft_consensus.cc:385] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:07.052170  8773 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:07.052851  8773 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.053794  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:07.054682  8650 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 2: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:01:07.055666  8773 ts_tablet_manager.cc:1434] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.000s
I20260709 13:01:07.058030  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:07.059188  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:07.062023  8773 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:07.063944  9206 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:01:07.065552  8696 leader_election.cc:343] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.066534  8696 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:07.067816  9241 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:07.068301  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:01:07.070350  8696 leader_election.cc:343] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.081384  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:01:07.082687  8549 leader_election.cc:343] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:07.084847  8549 leader_election.cc:343] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.087776  8774 tablet_bootstrap.cc:654] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
W20260709 13:01:07.088490  8549 leader_election.cc:343] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.092613  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:07.093655  9139 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:01:07.094017  8549 leader_election.cc:343] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:01:07.095142  8549 leader_election.cc:343] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.100454  9147 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.100999  9147 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.105376  9147 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.101331  8579 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.107331  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:07.108131  8727 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 0.
I20260709 13:01:07.110417  8625 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:07.112021  9147 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:01:07.112531  9147 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:07.112983  9147 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.113929  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:07.115233  9141 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:07.116080  8622 leader_election.cc:343] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.115716  9141 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.119306  8773 tablet_bootstrap.cc:654] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:07.119668  9141 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.120627  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:07.121034  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:07.122655  8727 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 1: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:01:07.123881  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:07.124993  9143 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.125020  9210 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:07.142083  9143 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:01:07.154115  8774 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:07.152055  9147 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.158958  9139 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:07.160231  8774 ts_tablet_manager.cc:1403] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.124s	user 0.015s	sys 0.010s
I20260709 13:01:07.160207  9139 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.161474  9147 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.162559  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:07.163118  8727 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.163964  9139 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.159508  8773 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:07.165076  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:07.165474  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:01:07.166690  8622 leader_election.cc:343] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:01:07.165117  8773 ts_tablet_manager.cc:1403] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.103s	user 0.011s	sys 0.007s
I20260709 13:01:07.170274  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:07.171821  8549 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:07.171914  8773 raft_consensus.cc:359] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.172545  8773 raft_consensus.cc:385] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:07.172812  8773 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:07.173210  9139 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:07.173959  8773 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.174577  8727 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:07.176324  8625 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:01:07.176911  8773 ts_tablet_manager.cc:1434] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.002s
I20260709 13:01:07.177544  8774 raft_consensus.cc:359] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.178025  9142 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:01:07.177946  8773 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:07.178342  8774 raft_consensus.cc:385] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:07.178612  9142 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:07.182583  8774 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:07.183389  8774 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.183305  9142 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.185928  8774 ts_tablet_manager.cc:1434] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.003s
I20260709 13:01:07.187071  8774 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Bootstrap starting.
I20260709 13:01:07.190748  8481 catalog_manager.cc:5697] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:07.193181  8774 tablet_bootstrap.cc:654] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:07.203753  8773 tablet_bootstrap.cc:654] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:07.218387  8479 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:07.244153  8479 catalog_manager.cc:5697] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:07.251398  9145 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:07.266173  9145 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.278954  8774 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: No bootstrap required, opened a new log
I20260709 13:01:07.279497  8774 ts_tablet_manager.cc:1403] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Time spent bootstrapping tablet: real 0.093s	user 0.041s	sys 0.034s
I20260709 13:01:07.282502  8774 raft_consensus.cc:359] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.283536  8774 raft_consensus.cc:385] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:07.283862  8774 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Initialized, Role: FOLLOWER
I20260709 13:01:07.284713  8774 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.287101  8774 ts_tablet_manager.cc:1434] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 13:01:07.292608  8773 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:07.293111  8773 ts_tablet_manager.cc:1403] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.115s	user 0.008s	sys 0.005s
I20260709 13:01:07.293885  8480 catalog_manager.cc:5697] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:07.295854  8773 raft_consensus.cc:359] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.296540  8773 raft_consensus.cc:385] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:07.296810  8773 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:07.297575  8773 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.299424  8773 ts_tablet_manager.cc:1434] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:01:07.300331  8773 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:07.301594  8480 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:07.305747  8480 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:07.308892  8773 tablet_bootstrap.cc:654] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:07.313781  8773 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:07.314235  8773 ts_tablet_manager.cc:1403] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 13:01:07.318832  8773 raft_consensus.cc:359] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.319602  8773 raft_consensus.cc:385] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:07.319871  8773 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:07.320619  8773 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.322801  8773 ts_tablet_manager.cc:1434] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:01:07.323779  8773 tablet_bootstrap.cc:492] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Bootstrap starting.
I20260709 13:01:07.329792  8773 tablet_bootstrap.cc:654] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Neither blocks nor log segments found. Creating new log.
I20260709 13:01:07.341907  8481 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:07.348232  8773 tablet_bootstrap.cc:492] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: No bootstrap required, opened a new log
I20260709 13:01:07.348903  8773 ts_tablet_manager.cc:1403] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Time spent bootstrapping tablet: real 0.025s	user 0.006s	sys 0.010s
I20260709 13:01:07.360641  8773 raft_consensus.cc:359] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.361572  8773 raft_consensus.cc:385] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:01:07.361874  8773 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Initialized, Role: FOLLOWER
I20260709 13:01:07.362504  8773 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.363940  8657 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.366114  8773 ts_tablet_manager.cc:1434] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.004s
I20260709 13:01:07.399082  9239 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.401199  8582 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.417284  9213 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:07.427205  8582 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:01:07.434628  9241 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:07.473563  8814 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.511409  8657 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.519481  9136 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.521229  8650 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.526314  8659 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:01:07.528571  9241 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.541173  9136 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:07.549965  9147 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.550555  9147 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.552685  9034 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.558051  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:07.561766  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:07.562436  8726 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:01:07.560966  9147 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.567003  8625 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:07.576880  9147 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:07.583573  9147 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:07.584729  9147 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:07.599954  9139 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.604488  9239 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:07.643841  8581 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.655027  9147 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.657944  9147 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.659210  8651 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:07.659835  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:07.663056  9136 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.664268  9139 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:07.664657  9139 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:07.665589  9139 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.667269  9139 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:07.672767  9239 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:07.673331  9239 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.675971  9239 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.677796  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:07.678563  8728 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:01:07.661989  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:07.680228  8550 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:07.680301  8625 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:07.681171  9239 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:07.681572  9239 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:07.681951  9239 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:07.681679  9147 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:01:07.682474  9142 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:07.683599  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:07.687809  8651 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:01:07.689268  9239 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.690472  8580 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:01:07.692476  9143 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:07.693316  9143 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:07.693662  9143 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:07.694445  9143 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.695611  9143 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:07.701999  9239 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:07.703692  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:07.704492  8728 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:07.705798  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:07.710431  8650 raft_consensus.cc:2410] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:01:07.721606  8728 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:07.723093  8550 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:07.723907  9239 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:07.724404  9239 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:07.725186  9239 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:07.745276  8481 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:08.129261  8726 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:01:08.130926  9139 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:08.144467  8651 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 13:01:08.145028  8651 pending_rounds.cc:85] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 0
I20260709 13:01:08.145501  8651 pending_rounds.cc:107] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:01:08.145839  8651 raft_consensus.cc:2889] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:08.148439  9139 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:08.232486  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:01:08.276378  9285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:01:08.277860  9286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:01:08.283114  9288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:01:08.284569  7481 server_base.cc:1061] running on GCE node
I20260709 13:01:08.290601  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:01:08.291006  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:01:08.291270  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602068291249 us; error 0 us; skew 500 ppm
I20260709 13:01:08.292101  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:01:08.296064  7481 webserver.cc:533] Webserver started at http://127.7.78.68:36359/ using document root <none> and password file <none>
I20260709 13:01:08.296682  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:01:08.297477  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:01:08.298023  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:01:08.299998  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-3-root/instance:
uuid: "453e644bad5d410ab9be85b353e28dc2"
format_stamp: "Formatted at 2026-07-09 13:01:08 on dist-test-slave-50cs"
I20260709 13:01:08.308745  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:01:08.315009  9293 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:01:08.316159  7481 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.003s
I20260709 13:01:08.316529  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-3-root
uuid: "453e644bad5d410ab9be85b353e28dc2"
format_stamp: "Formatted at 2026-07-09 13:01:08 on dist-test-slave-50cs"
I20260709 13:01:08.316799  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:01:08.344118  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:01:08.346220  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:01:08.349992  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:01:08.355353  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:01:08.355779  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20260709 13:01:08.356223  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:01:08.356536  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:01:08.582533  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.68:33133
I20260709 13:01:08.584081  9355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.68:33133 every 8 connection(s)
I20260709 13:01:08.590222  7481 auto_leader_rebalancer-test.cc:384] add a tserver: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:08.615532  9356 heartbeater.cc:344] Connected to a master server at 127.7.78.126:44983
I20260709 13:01:08.616132  9356 heartbeater.cc:461] Registering TS with master...
I20260709 13:01:08.617245  9356 heartbeater.cc:507] Master 127.7.78.126:44983 requested a full tablet report, sending...
I20260709 13:01:08.619743  8480 ts_manager.cc:194] Registered new tserver with Master: 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:08.621186  8480 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47036
I20260709 13:01:09.112833  8877 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:09.113593  8877 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:09.116070  8877 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:09.117049  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:09.117365  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:09.117951  8579 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:09.118070  8651 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 2: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:01:09.119017  8696 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:09.120653  9156 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:09.121035  9156 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:09.121435  9156 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:09.128355  9156 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:09.131578  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:09.131572  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:09.133792  8651 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 2: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:01:09.135032  8582 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:09.132130  9156 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:09.141135  8579 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:09.142148  8726 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.143126  8696 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:09.145500  9156 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:09.146019  9156 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:09.146811  9156 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:09.150533  8651 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.151537  9139 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:09.163802  9139 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:09.211227  8548 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0337bfcbeb3b4e15b96669c8d3ca4692. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:09.214361  8480 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:09.215015  9145 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.216465  8733 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.240046  8479 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:09.238993  8651 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.267751  8659 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:09.282933  8726 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.291612  8582 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.294389  9361 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:09.302239  9141 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:09.316347  9141 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.318429  8579 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.329088  8726 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.349579  8659 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:09.359532  9141 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:09.362452  9141 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:09.366854  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:09.367748  8579 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:09.371963  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:09.374550  9141 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:09.381739  9141 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:09.382262  9141 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:09.382678  9141 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:09.390419  8623 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98c31f9999e84270898dfb87c21cbc7c. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:01:09.427181  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 64475.3us
I20260709 13:01:09.429728  8733 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.429675  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:09.460228  9381 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:09.485867  8623 consensus_peers.cc:597] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77d6200607074a5ebd3e4ae470a2d6ed. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:09.468724  9141 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:09.486565  8480 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.488921  9141 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:09.491226  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:01:09.524659  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 67260.1us
I20260709 13:01:09.540280  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:09.540885  8579 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:09.540870  8580 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.551594  8726 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.560585  8579 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:01:09.550493  9383 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.574913  9142 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:09.597708  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:09.603312  9379 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:09.603845  9379 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:09.604660  9379 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:09.609094  8480 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.626057  9356 heartbeater.cc:499] Master 127.7.78.126:44983 was elected leader, sending a full tablet report...
I20260709 13:01:09.638684  9388 ts_tablet_manager.cc:933] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:09.678171  9388 tablet_copy_client.cc:323] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:09.697321  8579 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.715121  8480 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:09.732013  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44926.4us
W20260709 13:01:09.739773  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57985.5us
I20260709 13:01:09.750813  8650 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:09.751423  8726 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:09.756219  8594 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 0337bfcbeb3b4e15b96669c8d3ca4692 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:09.756915  8594 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 0337bfcbeb3b4e15b96669c8d3ca4692 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-0337bfcbeb3b4e15b96669c8d3ca4692
I20260709 13:01:09.768755  8579 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:09.768847  8581 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.770880  9213 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:09.772483  9395 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:01:09.773217  9395 consensus_queue.cc:1243] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 13:01:09.773880  9213 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:09.774334  9213 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:09.774545  8594 tablet_copy_source_session.cc:215] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:09.775240  8651 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.775246  9213 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:09.776684  9213 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:09.779181  9388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0337bfcbeb3b4e15b96669c8d3ca4692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:09.787307  9395 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:09.808115  8697 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 266236cfcd824409b757505321b36d81. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:09.814107  9388 tablet_copy_client.cc:806] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:09.813462  9393 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.814828  9388 tablet_copy_client.cc:670] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:09.821061  9388 tablet_copy_client.cc:538] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:09.838600  9388 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:09.844981  8651 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.856643  8480 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:09.856859  8579 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:09.921928  9388 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:09.922940  9388 tablet_bootstrap.cc:492] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:09.923588  9388 ts_tablet_manager.cc:1403] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.085s	user 0.049s	sys 0.021s
I20260709 13:01:09.927472  9403 ts_tablet_manager.cc:933] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:09.931833  9388 raft_consensus.cc:359] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:09.932832  9388 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:09.933539  9388 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:09.936065  9403 tablet_copy_client.cc:323] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:09.941577  9388 ts_tablet_manager.cc:1434] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.001s
I20260709 13:01:09.957389  8594 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-0337bfcbeb3b4e15b96669c8d3ca4692 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:09.957984  8594 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-0337bfcbeb3b4e15b96669c8d3ca4692 on tablet 0337bfcbeb3b4e15b96669c8d3ca4692 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:09.961098  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet 98c31f9999e84270898dfb87c21cbc7c from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:09.961861  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet 98c31f9999e84270898dfb87c21cbc7c from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-98c31f9999e84270898dfb87c21cbc7c
I20260709 13:01:09.971002  8669 tablet_copy_source_session.cc:215] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:09.974574  9403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98c31f9999e84270898dfb87c21cbc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:09.992390  9403 tablet_copy_client.cc:806] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:09.993026  9403 tablet_copy_client.cc:670] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:09.997452  9403 tablet_copy_client.cc:538] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:10.006458  9403 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:10.037215  9403 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:10.038143  9403 tablet_bootstrap.cc:492] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:10.038813  9403 ts_tablet_manager.cc:1403] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.033s	user 0.025s	sys 0.004s
I20260709 13:01:10.041555  8582 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:10.042210  9403 raft_consensus.cc:359] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:10.043045  9403 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:10.043676  9403 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:10.046322  9361 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:10.058724  9403 ts_tablet_manager.cc:1434] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 13:01:10.061625  8669 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-98c31f9999e84270898dfb87c21cbc7c received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:10.062105  8669 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-98c31f9999e84270898dfb87c21cbc7c on tablet 98c31f9999e84270898dfb87c21cbc7c with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:10.081377  9403 ts_tablet_manager.cc:933] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:10.097069  8728 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:01:10.097543  8728 pending_rounds.cc:85] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 1
I20260709 13:01:10.097782  8728 pending_rounds.cc:107] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:01:10.098026  8728 raft_consensus.cc:2889] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:10.102332  9379 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:10.111891  9403 tablet_copy_client.cc:323] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:10.115607  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet 77d6200607074a5ebd3e4ae470a2d6ed from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:10.116178  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet 77d6200607074a5ebd3e4ae470a2d6ed from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-77d6200607074a5ebd3e4ae470a2d6ed
I20260709 13:01:10.129334  8669 tablet_copy_source_session.cc:215] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:10.134047  9403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d6200607074a5ebd3e4ae470a2d6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:10.145479  9331 raft_consensus.cc:1217] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:10.169932  9403 tablet_copy_client.cc:806] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:10.170658  9403 tablet_copy_client.cc:670] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:10.180706  9403 tablet_copy_client.cc:538] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:10.192682  9403 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
W20260709 13:01:10.243793  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49451.1us
I20260709 13:01:10.330538  9388 ts_tablet_manager.cc:933] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:10.342751  9388 tablet_copy_client.cc:323] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:10.430469  9331 raft_consensus.cc:1217] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 13:01:10.433537  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 81670.4us
W20260709 13:01:10.488299  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 126376us
W20260709 13:01:10.493443  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 51793us
W20260709 13:01:10.495615  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53082.6us
W20260709 13:01:10.567153  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66643.3us
I20260709 13:01:10.583895  9403 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:10.595366  9403 tablet_bootstrap.cc:492] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:10.596350  9403 ts_tablet_manager.cc:1403] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.404s	user 0.052s	sys 0.053s
W20260709 13:01:10.622252  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44609.4us
I20260709 13:01:10.621102  9403 raft_consensus.cc:359] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:10.626060  9403 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:10.626744  9403 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:10.635394  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 266236cfcd824409b757505321b36d81 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:10.636040  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 266236cfcd824409b757505321b36d81 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-266236cfcd824409b757505321b36d81
I20260709 13:01:10.636510  9403 ts_tablet_manager.cc:1434] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.040s	user 0.010s	sys 0.000s
I20260709 13:01:10.641906  8669 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-77d6200607074a5ebd3e4ae470a2d6ed received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:10.642369  8669 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-77d6200607074a5ebd3e4ae470a2d6ed on tablet 77d6200607074a5ebd3e4ae470a2d6ed with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:10.652958  8743 tablet_copy_source_session.cc:215] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:10.670845  9421 raft_consensus.cc:1064] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:10.676994  9388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266236cfcd824409b757505321b36d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:10.676373  9421 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:10.699486  9388 tablet_copy_client.cc:806] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:10.700157  9388 tablet_copy_client.cc:670] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:10.723918  9388 tablet_copy_client.cc:538] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:10.734220  9388 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:10.814711  9331 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.816639  9421 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:01:10.850026  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:10.957713  8727 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.967675  9420 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:10.989773  9385 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:10.991473  8727 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.005784  8651 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.037626  9331 raft_consensus.cc:1217] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:11.052143  9142 raft_consensus.cc:1064] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:11.122678  9388 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:11.123862  9388 tablet_bootstrap.cc:492] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:11.124629  9388 ts_tablet_manager.cc:1403] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.391s	user 0.027s	sys 0.009s
W20260709 13:01:11.126487  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:11.129246  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 55749.8us
W20260709 13:01:11.130015  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66615.5us
I20260709 13:01:11.133666  9142 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:11.134733  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 71588.9us
I20260709 13:01:11.140980  9388 raft_consensus.cc:359] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:11.141918  9388 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:11.143918  9136 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:11.156646  9388 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:11.158270  9331 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.159525  9388 ts_tablet_manager.cc:1434] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.001s
I20260709 13:01:11.192773  8743 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-266236cfcd824409b757505321b36d81 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:11.192315  8479 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.193275  8743 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-266236cfcd824409b757505321b36d81 on tablet 266236cfcd824409b757505321b36d81 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:11.204726  9330 raft_consensus.cc:1217] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:11.211387  8651 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.232697  8727 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.234076  9379 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:11.248735  8583 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.254873  9142 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:11.265317  9331 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.269717  9379 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:11.309266  9142 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.321795  9331 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.322816  8583 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.341338  8578 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:11.351616  8727 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.361389  9330 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.363703  9423 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.365715  8729 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.375068  9420 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:11.376001  9421 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.371541  8480 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.403932  8659 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:11.376150  9330 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.415632  8729 raft_consensus.cc:2955] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.422436  9331 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.426936  9418 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.442737  9142 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.452934  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0337bfcbeb3b4e15b96669c8d3ca4692 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:11.468374  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 98c31f9999e84270898dfb87c21cbc7c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:11.469815  8578 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.476279  8482 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:11.488715  9387 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.487864  9330 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.493120  8481 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:11.508728  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 0337bfcbeb3b4e15b96669c8d3ca4692 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:11.534472  9440 tablet_replica.cc:333] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:11.536396  8583 raft_consensus.cc:2955] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.544339  9440 raft_consensus.cc:2243] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:11.552706  9440 raft_consensus.cc:2272] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:11.559381  9440 ts_tablet_manager.cc:1916] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:11.584081  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet 98c31f9999e84270898dfb87c21cbc7c with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:11.603258  9440 ts_tablet_manager.cc:1929] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:11.603943  9440 log.cc:1199] T 0337bfcbeb3b4e15b96669c8d3ca4692 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/0337bfcbeb3b4e15b96669c8d3ca4692
I20260709 13:01:11.606763  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 0337bfcbeb3b4e15b96669c8d3ca4692 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:11.608271  9441 tablet_replica.cc:333] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:11.609543  9441 raft_consensus.cc:2243] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:11.609872  9418 raft_consensus.cc:1064] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:11.610292  9441 raft_consensus.cc:2272] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:11.612746  9418 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:11.628336  8727 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.628639  9330 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.635116  8578 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.637949  9418 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:11.640789  9142 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:11.622232  9441 ts_tablet_manager.cc:1916] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:11.657882  9424 raft_consensus.cc:1064] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:11.660408  9424 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:11.656680  9418 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260709 13:01:11.683516  9330 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.684993  9416 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:11.699673  8651 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.702970  9441 ts_tablet_manager.cc:1929] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:11.703534  9441 log.cc:1199] T 98c31f9999e84270898dfb87c21cbc7c P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/98c31f9999e84270898dfb87c21cbc7c
I20260709 13:01:11.703603  8576 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.707222  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 98c31f9999e84270898dfb87c21cbc7c (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:11.707336  9395 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:11.708189  8877 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:11.723640  9387 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.729651  8727 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.731505  8583 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.750336  8481 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.765758  9331 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.795511  8651 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:11.817771  8726 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.823681  9331 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.822645  9361 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:01:11.826462  9379 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.827178  9419 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.851487  8659 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:11.854692  8513 debug-util.cc:398] Leaking SignalData structure 0x7b0800091de0 after lost signal to thread 7484
W20260709 13:01:11.855746  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08000644c0 after lost signal to thread 8516
W20260709 13:01:11.856904  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08002f1b00 after lost signal to thread 8608
W20260709 13:01:11.857967  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08000d35a0 after lost signal to thread 8683
W20260709 13:01:11.859019  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff4e0 after lost signal to thread 8757
W20260709 13:01:11.860103  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08002f1ac0 after lost signal to thread 9355
I20260709 13:01:11.847801  8577 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.862821  8726 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.897217  9331 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.900477  9330 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.914902  9387 raft_consensus.cc:2955] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.923750  8479 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.928418  8482 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:11.942040  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 77d6200607074a5ebd3e4ae470a2d6ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:11.945340  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet 77d6200607074a5ebd3e4ae470a2d6ed with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:12.012409  9451 tablet_replica.cc:333] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:12.013679  9451 raft_consensus.cc:2243] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:12.014518  9451 raft_consensus.cc:2272] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:12.017982  9451 ts_tablet_manager.cc:1916] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:12.134459  8727 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:12.156924  9331 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:12.163077  9419 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:12.166292  9451 ts_tablet_manager.cc:1929] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:12.170013  9451 log.cc:1199] T 77d6200607074a5ebd3e4ae470a2d6ed P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/77d6200607074a5ebd3e4ae470a2d6ed
I20260709 13:01:12.172801  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet 77d6200607074a5ebd3e4ae470a2d6ed (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:12.182192  9395 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:12.201568  9331 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:12.203224  8583 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:12.254069  9416 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:12.268532  8583 raft_consensus.cc:2955] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:12.286286  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 266236cfcd824409b757505321b36d81 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:12.311499  8479 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:12.359967  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 266236cfcd824409b757505321b36d81 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:12.369006  9453 tablet_replica.cc:333] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:12.382437  9453 raft_consensus.cc:2243] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:12.383316  9453 raft_consensus.cc:2272] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:12.388265  9453 ts_tablet_manager.cc:1916] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:12.410967  9453 ts_tablet_manager.cc:1929] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:12.411437  9453 log.cc:1199] T 266236cfcd824409b757505321b36d81 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/266236cfcd824409b757505321b36d81
I20260709 13:01:12.413669  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 266236cfcd824409b757505321b36d81 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:01:13.094678  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 131365us
I20260709 13:01:13.532852  8727 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:13.610625  8697 consensus_peers.cc:597] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf63362bb9b4409299528319b7472b04. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:13.612263  8583 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:13.616219  9419 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:13.625555  8651 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:13.630096  9460 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:13.645268  9419 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.651298  8583 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.658273  8657 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.665378  8577 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:13.670131  8482 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:13.684913  8726 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:13.686715  9384 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:13.689741  8548 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d18196b7e714cb6a79ec64ea3d65803. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:13.690434  8650 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:13.693327  9459 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:13.709870  9454 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.711350  8727 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.729013  8651 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.747586  8657 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:13.761543  8481 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.775787  8577 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:13.776062  8727 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:13.780138  8623 consensus_peers.cc:597] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6ad38ee7735403281a14a504308a8ea. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:13.780954  9457 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:13.783311  9455 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:13.801616  9455 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.807027  8577 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.807880  8726 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.825354  8482 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.831871  8726 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:13.859658  8697 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 689ad8326d634dee8402f1352a76f0e2. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:01:13.860015  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001d87a0 after lost signal to thread 7484
W20260709 13:01:13.861191  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08000878e0 after lost signal to thread 8516
W20260709 13:01:13.862416  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800295fa0 after lost signal to thread 8608
W20260709 13:01:13.863427  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800283c20 after lost signal to thread 8683
I20260709 13:01:13.863552  8657 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:13.864440  8577 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:13.864513  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080030f5a0 after lost signal to thread 8757
I20260709 13:01:13.864979  9419 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:13.866166  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080030fea0 after lost signal to thread 9355
I20260709 13:01:13.868373  9419 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:13.894050  9416 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
W20260709 13:01:13.944001  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45370.2us
I20260709 13:01:13.957450  8657 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.977826  8481 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:13.981346  8582 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:14.096557  9480 ts_tablet_manager.cc:933] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:14.151356  9480 tablet_copy_client.cc:323] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:14.154229  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet bf63362bb9b4409299528319b7472b04 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:14.154863  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet bf63362bb9b4409299528319b7472b04 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-bf63362bb9b4409299528319b7472b04
I20260709 13:01:14.164009  8743 tablet_copy_source_session.cc:215] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:14.177985  9480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf63362bb9b4409299528319b7472b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:14.233752  9480 tablet_copy_client.cc:806] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:14.234418  9480 tablet_copy_client.cc:670] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:14.246608  9480 tablet_copy_client.cc:538] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:14.262825  9480 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:14.296046  9482 ts_tablet_manager.cc:933] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:14.298926  9483 ts_tablet_manager.cc:933] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:14.316336  9482 tablet_copy_client.cc:323] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:14.320621  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet c6ad38ee7735403281a14a504308a8ea from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:14.321353  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet c6ad38ee7735403281a14a504308a8ea from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-c6ad38ee7735403281a14a504308a8ea
I20260709 13:01:14.337913  9480 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:14.350288  9480 tablet_bootstrap.cc:492] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:14.351197  9480 ts_tablet_manager.cc:1403] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.089s	user 0.026s	sys 0.001s
I20260709 13:01:14.353713  8669 tablet_copy_source_session.cc:215] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:14.354789  9480 raft_consensus.cc:359] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.355944  9480 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:14.356834  9480 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.362443  9487 ts_tablet_manager.cc:933] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:14.379907  9487 tablet_copy_client.cc:323] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:14.382275  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 689ad8326d634dee8402f1352a76f0e2 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:14.383016  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 689ad8326d634dee8402f1352a76f0e2 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-689ad8326d634dee8402f1352a76f0e2
I20260709 13:01:14.487892  8743 tablet_copy_source_session.cc:215] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:14.542348  9482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ad38ee7735403281a14a504308a8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:14.546940  9483 tablet_copy_client.cc:323] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:14.549299  8594 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 4d18196b7e714cb6a79ec64ea3d65803 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:14.549952  8594 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 4d18196b7e714cb6a79ec64ea3d65803 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-4d18196b7e714cb6a79ec64ea3d65803
I20260709 13:01:14.555683  9487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689ad8326d634dee8402f1352a76f0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:14.558557  8594 tablet_copy_source_session.cc:215] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:14.563580  9483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d18196b7e714cb6a79ec64ea3d65803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:14.590905  9480 ts_tablet_manager.cc:1434] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.239s	user 0.071s	sys 0.117s
I20260709 13:01:14.593451  8743 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-bf63362bb9b4409299528319b7472b04 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:14.594038  8743 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-bf63362bb9b4409299528319b7472b04 on tablet bf63362bb9b4409299528319b7472b04 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:14.614162  9487 tablet_copy_client.cc:806] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:14.614986  9487 tablet_copy_client.cc:670] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:14.621485  9487 tablet_copy_client.cc:538] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:14.630565  9482 tablet_copy_client.cc:806] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:14.631294  9482 tablet_copy_client.cc:670] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:14.650887  9482 tablet_copy_client.cc:538] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:14.658202  9487 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:14.677700  9482 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:14.710515  9483 tablet_copy_client.cc:806] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:14.723502  9483 tablet_copy_client.cc:670] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:14.742528  9483 tablet_copy_client.cc:538] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:14.790954  9483 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:14.811739  9487 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:14.817873  9487 tablet_bootstrap.cc:492] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:14.818938  9487 ts_tablet_manager.cc:1403] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.162s	user 0.023s	sys 0.006s
I20260709 13:01:14.822927  9487 raft_consensus.cc:359] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.827016  9487 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:14.835737  9487 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.841883  9487 ts_tablet_manager.cc:1434] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 13:01:14.846823  8743 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-689ad8326d634dee8402f1352a76f0e2 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:14.847348  8743 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-689ad8326d634dee8402f1352a76f0e2 on tablet 689ad8326d634dee8402f1352a76f0e2 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:14.858841  9330 raft_consensus.cc:1217] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:14.877265  9483 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:14.878476  9483 tablet_bootstrap.cc:492] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:14.879302  9483 ts_tablet_manager.cc:1403] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.089s	user 0.024s	sys 0.001s
I20260709 13:01:14.884351  9483 raft_consensus.cc:359] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.893575  9483 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:14.894460  9483 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.859082  9482 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:14.909793  9482 tablet_bootstrap.cc:492] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:14.910178  9483 ts_tablet_manager.cc:1434] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.001s
I20260709 13:01:14.912139  8594 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-4d18196b7e714cb6a79ec64ea3d65803 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:14.912573  8594 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-4d18196b7e714cb6a79ec64ea3d65803 on tablet 4d18196b7e714cb6a79ec64ea3d65803 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:14.912753  9482 ts_tablet_manager.cc:1403] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.235s	user 0.024s	sys 0.004s
I20260709 13:01:14.916002  9482 raft_consensus.cc:359] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.917198  9482 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:14.918048  9482 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:14.931218  9482 ts_tablet_manager.cc:1434] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.004s
I20260709 13:01:14.943068  9330 raft_consensus.cc:1217] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:14.965756  8669 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-c6ad38ee7735403281a14a504308a8ea received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:14.966296  8669 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-c6ad38ee7735403281a14a504308a8ea on tablet c6ad38ee7735403281a14a504308a8ea with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:15.135594  9330 raft_consensus.cc:1217] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 13:01:15.205586  9330 raft_consensus.cc:1217] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:15.373728  9458 raft_consensus.cc:1064] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:15.376459  9458 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:15.396515  9330 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.406718  9458 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.398631  8651 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.401964  8576 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.411769  9419 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.413949  9419 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:15.516084  9424 raft_consensus.cc:1064] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:15.563175  9419 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.594910  9501 raft_consensus.cc:1064] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:15.632952  9501 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:15.613624  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47455.1us
I20260709 13:01:15.615902  9424 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:15.643186  8650 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.648454  9331 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:15.650622  9384 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:15.660470  9455 raft_consensus.cc:1064] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:15.663939  9455 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:15.693789  8650 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 13:01:15.703305  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46474.7us
I20260709 13:01:15.723945  9502 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.728752  8576 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.747095  8650 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.748149  8578 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.764874  9329 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.769444  9329 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.783347  9458 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.785683  9498 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 13:01:15.795626  8726 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:15.778404  8479 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.804774  9424 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:15.817742  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 60395.7us
I20260709 13:01:15.824426  9503 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.827721  8579 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.837373  9455 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.890486  9500 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.906636  9510 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.912313  9507 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.912431  9456 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.912473  8726 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:15.926510  9512 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:15.931491  9330 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.945678  9329 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.948447  8579 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.964076  8576 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.970142  8651 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.970849  8657 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:15.985803  8479 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:16.020682  9328 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.033103  8482 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:16.051424  8578 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:16.073005  8727 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.013154  8481 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:16.190743  8728 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.208187  9329 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:16.179854  9331 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.251493  9523 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
W20260709 13:01:16.286425  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45855.6us
I20260709 13:01:16.380380  9523 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.385231  9329 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:16.401934  9532 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.279s	user 0.003s	sys 0.001s
W20260709 13:01:16.405916  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 201445us
I20260709 13:01:16.404573  8731 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:16.441233  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 332907us
I20260709 13:01:16.442509  9329 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:16.446434  9514 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:16.464687  8657 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:16.466099  9419 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.467820  9329 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.473755  8479 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:16.482336  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 689ad8326d634dee8402f1352a76f0e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:16.484160  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4d18196b7e714cb6a79ec64ea3d65803 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:16.484231  8581 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:16.493968  9419 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:16.504834  8581 raft_consensus.cc:2955] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:16.611385  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66709.8us
W20260709 13:01:17.159788  8581 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.653s	user 0.012s	sys 0.011s
I20260709 13:01:17.206125  9541 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:17.237617  9541 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:17.260603  9541 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:17.264632  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:17.343890  8657 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:17.344959  8482 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:17.398303  9328 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:17.401371  8728 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:17.401523  9512 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:17.402971  9538 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:17.438887  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:17.439651  9553 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:17.448493  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d18196b7e714cb6a79ec64ea3d65803 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:17.479835  9560 tablet_replica.cc:333] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:17.481274  9538 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:17.481904  9538 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:17.486465  9540 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.488441  9538 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:17.488617  8728 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.489562  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:17.491314  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:17.493983  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:17.496033  8625 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:17.510139  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:17.512984  9541 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:17.521713  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 689ad8326d634dee8402f1352a76f0e2 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:17.521253  8728 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:17.536892  9560 raft_consensus.cc:2243] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:17.537834  9560 raft_consensus.cc:2272] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:17.538705  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c6ad38ee7735403281a14a504308a8ea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:17.540861  9560 ts_tablet_manager.cc:1916] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:17.549039  9564 tablet_replica.cc:333] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:17.550980  8655 raft_consensus.cc:2955] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.550247  9564 raft_consensus.cc:2243] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:17.563691  8482 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.571698  9329 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:17.575716  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet c6ad38ee7735403281a14a504308a8ea with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:17.577673  9557 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:17.580391  8658 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:17.586131  9329 raft_consensus.cc:2955] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.587406  9564 raft_consensus.cc:2272] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:17.587803  9547 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:17.595858  9564 ts_tablet_manager.cc:1916] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:17.605715  9512 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:17.627774  9548 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.629942  9330 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.647090  9566 tablet_replica.cc:333] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:17.662124  8658 raft_consensus.cc:2955] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:17.671090  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet bf63362bb9b4409299528319b7472b04 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:17.674448  9560 ts_tablet_manager.cc:1929] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 13:01:17.674933  9560 log.cc:1199] T 4d18196b7e714cb6a79ec64ea3d65803 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/4d18196b7e714cb6a79ec64ea3d65803
I20260709 13:01:17.682790  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 4d18196b7e714cb6a79ec64ea3d65803 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:17.689817  9566 raft_consensus.cc:2243] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:17.690774  9566 raft_consensus.cc:2272] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:17.695060  9566 ts_tablet_manager.cc:1916] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:17.741153  8481 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:17.761500  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62006.8us
I20260709 13:01:17.799180  9564 ts_tablet_manager.cc:1929] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:17.799707  9564 log.cc:1199] T 689ad8326d634dee8402f1352a76f0e2 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/689ad8326d634dee8402f1352a76f0e2
I20260709 13:01:17.811224  9566 ts_tablet_manager.cc:1929] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:17.813831  9566 log.cc:1199] T c6ad38ee7735403281a14a504308a8ea P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/c6ad38ee7735403281a14a504308a8ea
I20260709 13:01:17.805164  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 689ad8326d634dee8402f1352a76f0e2 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:17.817492  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet bf63362bb9b4409299528319b7472b04 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:17.822117  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet c6ad38ee7735403281a14a504308a8ea (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:17.822837  9566 tablet_replica.cc:333] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:17.824920  9566 raft_consensus.cc:2243] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:01:17.829648  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62052.7us
I20260709 13:01:17.832221  9566 raft_consensus.cc:2272] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:17.850932  9566 ts_tablet_manager.cc:1916] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 13:01:17.882462  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 185515us
I20260709 13:01:17.937049  9566 ts_tablet_manager.cc:1929] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:17.957831  9566 log.cc:1199] T bf63362bb9b4409299528319b7472b04 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/bf63362bb9b4409299528319b7472b04
I20260709 13:01:17.961648  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet bf63362bb9b4409299528319b7472b04 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:01:18.195374  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08002cf740 after lost signal to thread 7484
W20260709 13:01:18.196465  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08002eda40 after lost signal to thread 8516
W20260709 13:01:18.197695  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08003023c0 after lost signal to thread 8608
W20260709 13:01:18.198755  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08002fd2c0 after lost signal to thread 8683
W20260709 13:01:18.199836  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08002ac760 after lost signal to thread 8757
W20260709 13:01:18.200959  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08003281c0 after lost signal to thread 9355
I20260709 13:01:18.995515  8576 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.017573  8731 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:19.019436  9571 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:19.032231  8548 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eed0f0ed87f74c1fb8320d38bceeac72. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:19.036299  8650 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:19.038947  9571 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:19.066895  9571 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.070626  8732 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.104442  8481 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.115854  8731 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.119321  8658 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.161198  8658 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:19.163018  8576 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:19.164424  9574 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:19.170521  9551 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:19.173367  8697 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ed4dfb6b4d146c3bb0c1c3677d1b39b. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:19.207218  9574 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.214231  8578 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.225637  8658 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.266196  8479 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
W20260709 13:01:19.376638  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 97547.8us
W20260709 13:01:19.392007  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 102416us
I20260709 13:01:19.435523  8731 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.481972  8655 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:19.482406  9591 ts_tablet_manager.cc:933] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
W20260709 13:01:19.483139  8697 consensus_peers.cc:597] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd0fc940d95d4b118d3e2564ed48b09e. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:19.484011  8579 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:19.484151  9589 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:19.486101  9557 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:19.498027  9591 tablet_copy_client.cc:323] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:19.500262  9575 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.502120  8594 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet eed0f0ed87f74c1fb8320d38bceeac72 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:19.502715  8594 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet eed0f0ed87f74c1fb8320d38bceeac72 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-eed0f0ed87f74c1fb8320d38bceeac72
I20260709 13:01:19.512584  8579 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.523500  8655 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.523667  8594 tablet_copy_source_session.cc:215] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:19.528226  8479 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:19.531670  9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed0f0ed87f74c1fb8320d38bceeac72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:19.554764  8658 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.572552  8731 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:19.574404  9457 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:19.576066  8579 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:19.578694  9457 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:01:19.581560  8623 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2adcf8a687b54060b644e8c691d8772f. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:19.586283  9596 ts_tablet_manager.cc:933] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:19.645533  9591 tablet_copy_client.cc:806] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:19.646351  9591 tablet_copy_client.cc:670] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:19.646198  9572 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.649027  8578 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.650849  9591 tablet_copy_client.cc:538] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:19.657810  9596 tablet_copy_client.cc:323] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:19.660272  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:19.660897  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-4ed4dfb6b4d146c3bb0c1c3677d1b39b
I20260709 13:01:19.662724  8732 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:19.668650  8743 tablet_copy_source_session.cc:215] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:19.684840  8479 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:19.662819  9591 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:19.703554  9596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:19.737908  9596 tablet_copy_client.cc:806] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:19.738585  9596 tablet_copy_client.cc:670] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:19.753422  9596 tablet_copy_client.cc:538] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:19.764353  9596 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:19.834501  9591 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:19.846246  9591 tablet_bootstrap.cc:492] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:19.847111  9591 ts_tablet_manager.cc:1403] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.185s	user 0.016s	sys 0.018s
I20260709 13:01:19.856142  9591 raft_consensus.cc:359] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.870147  9596 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:19.871228  9596 tablet_bootstrap.cc:492] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:19.872007  9596 ts_tablet_manager.cc:1403] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.108s	user 0.018s	sys 0.005s
I20260709 13:01:19.873359  9591 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:19.874230  9591 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.878404  9596 raft_consensus.cc:359] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.879603  9596 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:19.880371  9596 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:19.894989  9591 ts_tablet_manager.cc:1434] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.047s	user 0.008s	sys 0.001s
I20260709 13:01:19.901139  8594 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-eed0f0ed87f74c1fb8320d38bceeac72 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:19.901650  8594 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-eed0f0ed87f74c1fb8320d38bceeac72 on tablet eed0f0ed87f74c1fb8320d38bceeac72 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:19.935088  9596 ts_tablet_manager.cc:1434] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.063s	user 0.012s	sys 0.000s
I20260709 13:01:19.938719  8743 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-4ed4dfb6b4d146c3bb0c1c3677d1b39b received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:19.939239  8743 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-4ed4dfb6b4d146c3bb0c1c3677d1b39b on tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:19.947335  9330 raft_consensus.cc:1217] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:20.102974  9596 ts_tablet_manager.cc:933] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:20.113339  9591 ts_tablet_manager.cc:933] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:20.117158  9596 tablet_copy_client.cc:323] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:20.120671  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet dd0fc940d95d4b118d3e2564ed48b09e from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:20.121299  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet dd0fc940d95d4b118d3e2564ed48b09e from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-dd0fc940d95d4b118d3e2564ed48b09e
I20260709 13:01:20.130595  8743 tablet_copy_source_session.cc:215] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:20.139755  9596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd0fc940d95d4b118d3e2564ed48b09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:20.142254  9330 raft_consensus.cc:1217] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:20.167375  9591 tablet_copy_client.cc:323] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:20.173034  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet 2adcf8a687b54060b644e8c691d8772f from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:20.173764  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet 2adcf8a687b54060b644e8c691d8772f from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-2adcf8a687b54060b644e8c691d8772f
I20260709 13:01:20.172271  9596 tablet_copy_client.cc:806] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:20.175624  9596 tablet_copy_client.cc:670] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:20.180054  9596 tablet_copy_client.cc:538] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:20.190687  9596 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:20.237195  8669 tablet_copy_source_session.cc:215] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:20.239168  9596 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:20.240411  9596 tablet_bootstrap.cc:492] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:20.241546  9596 ts_tablet_manager.cc:1403] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.051s	user 0.030s	sys 0.004s
I20260709 13:01:20.249758  9591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2adcf8a687b54060b644e8c691d8772f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:20.244954  9596 raft_consensus.cc:359] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:20.254087  9596 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:20.254859  9596 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:20.257682  9596 ts_tablet_manager.cc:1434] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:01:20.265763  8743 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-dd0fc940d95d4b118d3e2564ed48b09e received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:20.266418  8743 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-dd0fc940d95d4b118d3e2564ed48b09e on tablet dd0fc940d95d4b118d3e2564ed48b09e with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:20.368436  9576 raft_consensus.cc:1064] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
W20260709 13:01:20.370263  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 92349.4us
I20260709 13:01:20.374765  9576 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:20.408298  8731 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:20.408231  9330 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:20.409873  9571 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:20.410585  8658 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:20.411944  9571 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:20.427140  9591 tablet_copy_client.cc:806] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:20.427947  9591 tablet_copy_client.cc:670] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:20.437552  9591 tablet_copy_client.cc:538] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:20.454468  9571 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:20.468686  9591 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:20.478608  9576 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.483726  8731 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.496686  9329 raft_consensus.cc:1217] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:20.510381  9330 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.520030  8658 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.521080  8482 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.571888  9589 raft_consensus.cc:1064] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:20.573747  9591 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:20.575189  9591 tablet_bootstrap.cc:492] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:20.575893  9591 ts_tablet_manager.cc:1403] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.107s	user 0.024s	sys 0.004s
I20260709 13:01:20.579212  9589 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:20.585340  9591 raft_consensus.cc:359] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:20.588971  9591 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:20.597227  9591 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:20.603782  9329 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:20.606870  9591 ts_tablet_manager.cc:1434] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260709 13:01:20.607635  8578 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:20.609506  8655 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:20.609782  9329 raft_consensus.cc:1217] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 13:01:20.611037  8669 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-2adcf8a687b54060b644e8c691d8772f received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:20.611711  8669 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-2adcf8a687b54060b644e8c691d8772f on tablet 2adcf8a687b54060b644e8c691d8772f with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:20.612170  9557 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260709 13:01:20.620359  9575 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:20.622920  9588 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:01:20.659171  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47008.6us
I20260709 13:01:20.695403  9618 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.709568  8581 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:20.715042  8655 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.743716  8578 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.767203  9329 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.783586  8478 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:20.791635  9330 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:20.798350  9622 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:20.802222  8658 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:20.805595  9623 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:20.814371  9608 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.815799  9330 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.824249  8655 raft_consensus.cc:2955] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.855481  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet eed0f0ed87f74c1fb8320d38bceeac72 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:20.858074  8481 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.863641  8732 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:20.882133  9329 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:20.884330  9624 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:20.893011  9618 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.895388  9330 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.915462  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet eed0f0ed87f74c1fb8320d38bceeac72 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:20.904273  8578 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:20.921185  9589 raft_consensus.cc:1064] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:20.924005  9589 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:20.927719  9624 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:20.932289  8578 raft_consensus.cc:2955] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:20.934952  9625 tablet_replica.cc:333] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:20.936323  9625 raft_consensus.cc:2243] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:20.937078  9625 raft_consensus.cc:2272] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:20.940317  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:20.947400  9625 ts_tablet_manager.cc:1916] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:20.954473  8479 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:20.974978  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:21.026630  8655 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:21.028043  9330 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:21.029628  9588 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:21.030498  9557 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:21.037858  9625 ts_tablet_manager.cc:1929] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:21.041988  9625 log.cc:1199] T eed0f0ed87f74c1fb8320d38bceeac72 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/eed0f0ed87f74c1fb8320d38bceeac72
I20260709 13:01:21.043890  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet eed0f0ed87f74c1fb8320d38bceeac72 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:21.050467  9627 tablet_replica.cc:333] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:21.051756  9627 raft_consensus.cc:2243] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:21.066720  8578 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:21.063771  9627 raft_consensus.cc:2272] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:21.079715  9575 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:21.085992  9627 ts_tablet_manager.cc:1916] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:21.117314  9575 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.133605  9329 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.124297  8658 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.156014  8479 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.164088  8578 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.176734  8732 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:21.178251  9627 ts_tablet_manager.cc:1929] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:21.178900  9627 log.cc:1199] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/4ed4dfb6b4d146c3bb0c1c3677d1b39b
I20260709 13:01:21.188114  9329 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:21.190070  9557 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:21.198091  9575 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.199991  9330 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.226929  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet dd0fc940d95d4b118d3e2564ed48b09e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:21.251353  8479 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.263329  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet dd0fc940d95d4b118d3e2564ed48b09e with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:21.264678  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 4ed4dfb6b4d146c3bb0c1c3677d1b39b (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:21.270092  8658 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:21.277576  9618 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:21.287936  9601 raft_consensus.cc:1064] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:21.292686  9634 tablet_replica.cc:333] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: stopping tablet replica
W20260709 13:01:21.293079  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 142352us
I20260709 13:01:21.293958  9634 raft_consensus.cc:2243] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:21.294710  9634 raft_consensus.cc:2272] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:21.305847  8658 raft_consensus.cc:2955] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.329932  9634 ts_tablet_manager.cc:1916] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:21.372822  9601 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:21.460510  8732 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:01:21.461491  9329 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:01:21.463435  9587 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:21.464573  9632 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:21.465382  8576 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:01:21.467655  9633 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:21.500468  9587 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.502765  8576 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.509668  8732 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.525779  9634 ts_tablet_manager.cc:1929] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:21.528513  9329 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.544674  8479 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:21.563382  9634 log.cc:1199] T dd0fc940d95d4b118d3e2564ed48b09e P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/dd0fc940d95d4b118d3e2564ed48b09e
I20260709 13:01:21.568073  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet dd0fc940d95d4b118d3e2564ed48b09e (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:21.573048  8657 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:21.583544  9330 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:21.588372  9601 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:21.591588  8578 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:21.596859  9587 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:21.603498  9601 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.606237  8576 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.610126  9330 raft_consensus.cc:2955] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.631948  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2adcf8a687b54060b644e8c691d8772f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:21.634364  8478 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:21.656719  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet 2adcf8a687b54060b644e8c691d8772f with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:21.839643  9640 tablet_replica.cc:333] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:21.841048  9640 raft_consensus.cc:2243] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:21.842199  9640 raft_consensus.cc:2272] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:21.930112  9640 ts_tablet_manager.cc:1916] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:21.972385  9640 ts_tablet_manager.cc:1929] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260709 13:01:21.972805  9640 log.cc:1199] T 2adcf8a687b54060b644e8c691d8772f P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/2adcf8a687b54060b644e8c691d8772f
I20260709 13:01:21.974265  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 2adcf8a687b54060b644e8c691d8772f (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:01:22.408917  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98217.4us
I20260709 13:01:22.890976  8650 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:22.909638  8623 consensus_peers.cc:597] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96ad7a91a5c14870885cd58a34393f82. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:22.911546  8576 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:22.920161  9642 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:22.934396  8732 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:22.936151  9642 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:23.055244  9642 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.064626  8732 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.143878  8582 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:23.153879  8576 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.164806  8481 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.181824  8733 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:23.183637  8548 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b06c2e2a7c644478ef688b47c11a07a. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:23.185304  9649 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:23.193164  8658 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:23.196882  9623 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:23.259912  9644 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.267592  8733 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.269856  8658 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.301942  8582 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:23.318763  8548 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ad2aa6046e746348b02a2571b7580f0. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:23.335294  8652 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:23.339268  9670 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:23.339385  8479 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:23.348668  8733 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:23.352077  9644 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:23.368808  9669 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.393617  9674 ts_tablet_manager.cc:933] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:23.397117  8652 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.406587  8732 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:23.412037  8733 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.474733  8481 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:23.555792  9674 tablet_copy_client.cc:323] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:23.560462  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet 96ad7a91a5c14870885cd58a34393f82 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:23.561532  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet 96ad7a91a5c14870885cd58a34393f82 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-96ad7a91a5c14870885cd58a34393f82
I20260709 13:01:23.576440  8652 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:23.576701  8669 tablet_copy_source_session.cc:215] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 13:01:23.576933  8697 consensus_peers.cc:597] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 162db4080f654c9fbd8c23faf68f6df3. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:23.578433  9653 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:23.578914  8582 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:23.580273  9653 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:23.587224  9674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ad7a91a5c14870885cd58a34393f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:23.617795  9575 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.623150  8652 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.629988  8582 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:23.679618  9674 tablet_copy_client.cc:806] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:23.680617  9674 tablet_copy_client.cc:670] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:23.680505  8479 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:23.686093  9674 tablet_copy_client.cc:538] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:23.738988  9674 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
W20260709 13:01:23.780730  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53587.9us
I20260709 13:01:23.795600  9674 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:23.796823  9674 tablet_bootstrap.cc:492] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:23.797668  9674 ts_tablet_manager.cc:1403] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.059s	user 0.015s	sys 0.019s
I20260709 13:01:23.798552  9679 ts_tablet_manager.cc:933] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:23.801252  9674 raft_consensus.cc:359] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:23.802879  9674 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:23.803674  9674 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:23.812748  9682 ts_tablet_manager.cc:933] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:23.812626  9679 tablet_copy_client.cc:323] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:23.829542  8594 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 4b06c2e2a7c644478ef688b47c11a07a from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:23.830243  8594 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 4b06c2e2a7c644478ef688b47c11a07a from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-4b06c2e2a7c644478ef688b47c11a07a
I20260709 13:01:23.840847  8594 tablet_copy_source_session.cc:215] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:23.853936  9682 tablet_copy_client.cc:323] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:23.855803  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 3ad2aa6046e746348b02a2571b7580f0 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:23.856400  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 3ad2aa6046e746348b02a2571b7580f0 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-3ad2aa6046e746348b02a2571b7580f0
I20260709 13:01:23.865518  9679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b06c2e2a7c644478ef688b47c11a07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:23.867259  8593 tablet_copy_source_session.cc:215] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:23.870733  9674 ts_tablet_manager.cc:1434] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.073s	user 0.005s	sys 0.033s
I20260709 13:01:23.875147  8669 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-96ad7a91a5c14870885cd58a34393f82 received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:23.875667  8669 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-96ad7a91a5c14870885cd58a34393f82 on tablet 96ad7a91a5c14870885cd58a34393f82 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:23.899343  9682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ad2aa6046e746348b02a2571b7580f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:23.980532  9329 raft_consensus.cc:1217] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:24.003965  9674 ts_tablet_manager.cc:933] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:24.020121  9674 tablet_copy_client.cc:323] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:24.022466  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 162db4080f654c9fbd8c23faf68f6df3 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:24.023136  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 162db4080f654c9fbd8c23faf68f6df3 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-162db4080f654c9fbd8c23faf68f6df3
I20260709 13:01:24.036598  8743 tablet_copy_source_session.cc:215] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:24.057276  9674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162db4080f654c9fbd8c23faf68f6df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:24.098963  9679 tablet_copy_client.cc:806] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:24.099762  9679 tablet_copy_client.cc:670] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:24.105341  9679 tablet_copy_client.cc:538] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:24.143796  9682 tablet_copy_client.cc:806] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:24.144596  9682 tablet_copy_client.cc:670] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:24.151110  9682 tablet_copy_client.cc:538] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:24.153059  9679 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:24.196015  9682 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:24.287477  9682 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:24.291055  9682 tablet_bootstrap.cc:492] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:24.292867  9682 ts_tablet_manager.cc:1403] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.097s	user 0.018s	sys 0.008s
I20260709 13:01:24.296875  9682 raft_consensus.cc:359] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:24.308187  9682 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:24.320825  9679 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:24.330192  9679 tablet_bootstrap.cc:492] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:24.314015  9682 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:24.335048  9674 tablet_copy_client.cc:806] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:24.338299  9674 tablet_copy_client.cc:670] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:24.340325  9679 ts_tablet_manager.cc:1403] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.188s	user 0.042s	sys 0.003s
I20260709 13:01:24.339499  9682 ts_tablet_manager.cc:1434] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.000s
I20260709 13:01:24.349677  9679 raft_consensus.cc:359] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:24.350657  9679 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:24.358930  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-3ad2aa6046e746348b02a2571b7580f0 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:24.359658  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-3ad2aa6046e746348b02a2571b7580f0 on tablet 3ad2aa6046e746348b02a2571b7580f0 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:24.351459  9679 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:24.367884  9679 ts_tablet_manager.cc:1434] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 13:01:24.369952  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-4b06c2e2a7c644478ef688b47c11a07a received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:24.370460  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-4b06c2e2a7c644478ef688b47c11a07a on tablet 4b06c2e2a7c644478ef688b47c11a07a with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:24.375434  9674 tablet_copy_client.cc:538] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:01:24.389923  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44689.7us
I20260709 13:01:24.561497  9658 raft_consensus.cc:1064] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:24.565124  9658 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:24.740592  9674 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
W20260709 13:01:24.758394  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 97725.4us
I20260709 13:01:24.798928  9329 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:24.802908  9329 raft_consensus.cc:1217] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:24.815435  9700 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
W20260709 13:01:24.892279  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 82063.2us
I20260709 13:01:24.925930  8731 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:24.942584  9700 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:24.954663  9328 raft_consensus.cc:1217] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 13:01:24.988241  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 105879us
W20260709 13:01:24.995637  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 108320us
W20260709 13:01:25.051498  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 171725us
I20260709 13:01:25.060645  9700 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.063086  9674 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:25.064112  9674 tablet_bootstrap.cc:492] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:25.064931  9674 ts_tablet_manager.cc:1403] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.325s	user 0.025s	sys 0.002s
I20260709 13:01:25.063733  8731 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.077273  9330 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.088200  9674 raft_consensus.cc:359] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:25.097976  9674 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:25.101112  9674 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:25.088490  8479 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.115134  9674 ts_tablet_manager.cc:1434] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.000s
I20260709 13:01:25.128981  8743 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-162db4080f654c9fbd8c23faf68f6df3 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:25.129613  8743 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-162db4080f654c9fbd8c23faf68f6df3 on tablet 162db4080f654c9fbd8c23faf68f6df3 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:25.191116  8582 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.206858  9704 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:25.224979  9330 raft_consensus.cc:1217] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:25.273823  8578 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.312533  8655 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:25.343849  9330 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:25.346539  8733 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:25.348357  9700 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:25.349820  9705 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:25.364341  9705 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.377352  9691 raft_consensus.cc:1064] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:25.383741  9691 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:25.397176  8733 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.401620  9330 raft_consensus.cc:2955] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.488312  8731 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.489682  9328 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.489993  9692 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:25.491580  9711 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:25.535442  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 96ad7a91a5c14870885cd58a34393f82 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:25.550088  8482 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.559697  8650 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.569562  9691 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.575395  8733 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.577797  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet 96ad7a91a5c14870885cd58a34393f82 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:25.605260  9722 tablet_replica.cc:333] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:25.607174  8481 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.621140  9693 raft_consensus.cc:1064] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:25.623854  9698 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:25.628922  9693 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:25.635079  9329 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.646641  9722 raft_consensus.cc:2243] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:01:25.649115  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48932us
I20260709 13:01:25.644578  8655 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.659288  9722 raft_consensus.cc:2272] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:25.691604  9722 ts_tablet_manager.cc:1916] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:25.698549  9703 raft_consensus.cc:1064] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:25.703750  9703 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:25.774277  9329 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.782809  9648 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:25.774518  8655 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.771754  8578 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:25.786139  8576 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.795433  9703 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:25.796667  9330 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:25.797088  9575 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:25.809221  9709 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:25.818078  8732 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.823381  9328 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.831970  8657 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:25.823922  9670 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:25.833456  9694 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:25.836959  8733 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 13:01:25.896526  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 67257.5us
I20260709 13:01:25.898077  9711 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:25.901702  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 72750.9us
I20260709 13:01:25.913347  9329 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.953014  8482 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:25.926602  9693 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 13:01:25.929666  9698 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
W20260709 13:01:26.033718  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 73282.2us
I20260709 13:01:26.013448  9722 ts_tablet_manager.cc:1929] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:26.039122  9722 log.cc:1199] T 96ad7a91a5c14870885cd58a34393f82 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/96ad7a91a5c14870885cd58a34393f82
I20260709 13:01:26.041729  8731 raft_consensus.cc:2955] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.043433  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet 96ad7a91a5c14870885cd58a34393f82 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:26.053356  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ad2aa6046e746348b02a2571b7580f0 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:26.063673  9728 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.065604  8579 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.074812  8655 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.079326  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3ad2aa6046e746348b02a2571b7580f0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:26.127251  9740 tablet_replica.cc:333] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:26.131312  9739 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.168203  9329 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.170354  9328 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.205310  8479 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.214584  9740 raft_consensus.cc:2243] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:26.215348  9740 raft_consensus.cc:2272] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:26.227437  9740 ts_tablet_manager.cc:1916] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:26.233566  8731 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.235821  8655 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.239867  8478 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.269917  8581 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:26.276506  9328 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:26.278340  9723 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:26.286504  9711 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.295394  8733 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:26.301515  8657 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:26.291946  9329 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.306631  9709 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:26.314105  9328 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:26.317724  8657 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:26.319718  8655 raft_consensus.cc:2955] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.320590  9737 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:01:26.327764  9728 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:26.344560  8482 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.350262  9699 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.358621  9330 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.357285  8657 raft_consensus.cc:2955] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.362991  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b06c2e2a7c644478ef688b47c11a07a with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:26.388227  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4b06c2e2a7c644478ef688b47c11a07a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:26.388150  9740 ts_tablet_manager.cc:1929] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:26.407491  9742 tablet_replica.cc:333] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:26.409667  8482 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:26.407136  9740 log.cc:1199] T 3ad2aa6046e746348b02a2571b7580f0 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/3ad2aa6046e746348b02a2571b7580f0
I20260709 13:01:26.418082  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 3ad2aa6046e746348b02a2571b7580f0 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:26.427827  9742 raft_consensus.cc:2243] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:26.428920  9742 raft_consensus.cc:2272] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:26.421110  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 162db4080f654c9fbd8c23faf68f6df3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:26.454634  9742 ts_tablet_manager.cc:1916] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:26.474340  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet 162db4080f654c9fbd8c23faf68f6df3 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:26.475344  9722 tablet_replica.cc:333] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:26.476459  9722 raft_consensus.cc:2243] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:26.477269  9722 raft_consensus.cc:2272] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:26.481650  9722 ts_tablet_manager.cc:1916] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:26.505717  9742 ts_tablet_manager.cc:1929] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:26.506295  9742 log.cc:1199] T 4b06c2e2a7c644478ef688b47c11a07a P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/4b06c2e2a7c644478ef688b47c11a07a
I20260709 13:01:26.507012  9722 ts_tablet_manager.cc:1929] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:26.507483  9722 log.cc:1199] T 162db4080f654c9fbd8c23faf68f6df3 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/162db4080f654c9fbd8c23faf68f6df3
I20260709 13:01:26.514472  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 4b06c2e2a7c644478ef688b47c11a07a (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:26.514910  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet 162db4080f654c9fbd8c23faf68f6df3 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:01:27.600740  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44162.1us
W20260709 13:01:27.744545  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63942.2us
I20260709 13:01:28.614079  8655 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:28.615252  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43988.4us
I20260709 13:01:28.628960  8731 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:28.631251  8578 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:28.632081  8623 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cde36bf9fcc5409e8a00b747fbfbfb06. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:28.632845  9704 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:28.633972  9757 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:28.654811  9757 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:28.657305  8726 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:28.667874  8582 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:28.674770  8578 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:28.699596  8478 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:28.704926  8655 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:28.705004  8726 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:28.705492  8548 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a389d19cf2c47d6b5873139d8cfad40. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:28.706831  9739 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:28.709280  9739 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:28.720295  9751 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:28.723980  8655 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:28.738765  8726 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:28.759120  8655 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:28.807096  8478 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:28.830633  8726 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:28.837500  8623 consensus_peers.cc:597] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3d38a98ca324ddbbb3ff25fb6ded965. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:28.838784  9757 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:28.840611  8578 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:28.846542  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:28.853876  9757 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:28.870934  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08002f0960 after lost signal to thread 7484
W20260709 13:01:28.872177  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff520 after lost signal to thread 8516
W20260709 13:01:28.873560  8513 debug-util.cc:398] Leaking SignalData structure 0x7b0800064b20 after lost signal to thread 8608
W20260709 13:01:28.874722  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08000642c0 after lost signal to thread 8683
W20260709 13:01:28.875866  8513 debug-util.cc:398] Leaking SignalData structure 0x7b080032b320 after lost signal to thread 8757
W20260709 13:01:28.877068  8513 debug-util.cc:398] Leaking SignalData structure 0x7b0800329e80 after lost signal to thread 9355
I20260709 13:01:29.039384  9757 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:29.048998  8731 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:29.080178  8578 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:29.089093  8651 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:29.094815  8478 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:29.125869  9771 ts_tablet_manager.cc:933] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:29.131978  8726 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:29.132687  8623 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04399390cf414886a3453981dc1f74aa. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:29.134423  8584 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:29.135818  9745 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:29.138243  9757 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:29.146086  9771 tablet_copy_client.cc:323] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:29.154341  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet cde36bf9fcc5409e8a00b747fbfbfb06 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:29.158228  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet cde36bf9fcc5409e8a00b747fbfbfb06 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-cde36bf9fcc5409e8a00b747fbfbfb06
I20260709 13:01:29.166781  8669 tablet_copy_source_session.cc:215] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:29.176719  9745 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:29.181510  9771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde36bf9fcc5409e8a00b747fbfbfb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:29.182891  8726 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:29.191357  8584 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:29.219255  9776 ts_tablet_manager.cc:933] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:29.218967  8478 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:29.230093  9771 tablet_copy_client.cc:806] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:29.238469  9771 tablet_copy_client.cc:670] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:29.245714  9776 tablet_copy_client.cc:323] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:29.247457  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 9a389d19cf2c47d6b5873139d8cfad40 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:29.248209  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 9a389d19cf2c47d6b5873139d8cfad40 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-9a389d19cf2c47d6b5873139d8cfad40
I20260709 13:01:29.251662  9771 tablet_copy_client.cc:538] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:29.257999  8593 tablet_copy_source_session.cc:215] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:29.262144  9776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a389d19cf2c47d6b5873139d8cfad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:29.276512  9771 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:29.283412  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:29.284808  9776 tablet_copy_client.cc:806] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:29.303634  8726 raft_consensus.cc:606] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:29.305081  9776 tablet_copy_client.cc:670] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:29.312523  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:29.313446  8726 raft_consensus.cc:606] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:29.317494  9776 tablet_copy_client.cc:538] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:29.361120  9748 raft_consensus.cc:993] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:29.361831  9754 raft_consensus.cc:1081] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:29.363771  9330 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:29.364338  9330 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:29.364640  9330 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.377645  9780 ts_tablet_manager.cc:933] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:29.378798  9330 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:29.382695  9776 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:29.395308  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:29.396051  8726 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:29.396365  8726 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:29.397070  8726 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:29.398224  8726 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.401085  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:29.401667  8651 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.414781  8578 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:29.415516  8578 raft_consensus.cc:606] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:29.420727  9728 raft_consensus.cc:993] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:29.421212  9728 raft_consensus.cc:1081] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:29.421501  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:29.422091  8728 raft_consensus.cc:606] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:29.425693  8650 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:29.426267  8650 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:29.426612  8650 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.427942  8651 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:29.429991  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:29.432492  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:29.433110  8728 raft_consensus.cc:606] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:29.433947  9780 tablet_copy_client.cc:323] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:29.440703  8650 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.447499  8726 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:29.450364  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:29.451937  8726 raft_consensus.cc:606] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:29.453069  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:29.453763  8584 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.459867  8650 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:29.439273  9330 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:29.462620  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:29.460624  8584 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:29.464762  8728 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:29.465080  8728 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:29.464267  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:29.468492  9783 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:29.468989  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:29.469884  8728 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.475039  8728 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.475920  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:29.476876  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet c3d38a98ca324ddbbb3ff25fb6ded965 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:29.479374  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet c3d38a98ca324ddbbb3ff25fb6ded965 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-c3d38a98ca324ddbbb3ff25fb6ded965
I20260709 13:01:29.485145  9752 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:29.485723  9752 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:29.486627  9752 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.524196  8669 tablet_copy_source_session.cc:215] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:29.529647  8728 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:29.550961  9780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d38a98ca324ddbbb3ff25fb6ded965. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:01:29.577838  9771 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.051s	user 0.002s	sys 0.000s
I20260709 13:01:29.576478  8482 catalog_manager.cc:5697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:29.554924  9783 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:29.594632  9771 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:29.594923  9780 tablet_copy_client.cc:806] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:29.595563  9780 tablet_copy_client.cc:670] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:29.595942  9771 tablet_bootstrap.cc:492] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:29.597944  9771 ts_tablet_manager.cc:1403] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.322s	user 0.028s	sys 0.008s
I20260709 13:01:29.586444  9783 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:29.621018  9787 ts_tablet_manager.cc:933] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:29.623486  9771 raft_consensus.cc:359] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:29.624377  9771 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:29.627750  9780 tablet_copy_client.cc:538] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:29.637948  9771 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:29.641440  9771 ts_tablet_manager.cc:1434] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.030s	user 0.004s	sys 0.002s
I20260709 13:01:29.644747  8669 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-cde36bf9fcc5409e8a00b747fbfbfb06 received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:29.645174  8669 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-cde36bf9fcc5409e8a00b747fbfbfb06 on tablet cde36bf9fcc5409e8a00b747fbfbfb06 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:29.665865  9780 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:29.664899  8482 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:29.672385  9748 raft_consensus.cc:993] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:29.672876  9748 raft_consensus.cc:1081] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:29.670136  9787 tablet_copy_client.cc:323] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:29.674885  8650 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:29.674968  8669 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet 04399390cf414886a3453981dc1f74aa from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:29.675503  8650 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:29.675539  8669 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet 04399390cf414886a3453981dc1f74aa from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-04399390cf414886a3453981dc1f74aa
I20260709 13:01:29.676002  8650 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.682745  8650 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.685737  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:29.686343  8726 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:29.686739  8726 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:29.684667  8650 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:29.688119  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:29.688638  8578 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.684072  8669 tablet_copy_source_session.cc:215] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:29.690403  8726 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.697847  8726 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.707760  9330 raft_consensus.cc:1217] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:29.709630  9787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04399390cf414886a3453981dc1f74aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:29.718014  8726 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:29.718014  8578 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:29.719322  8625 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:29.720148  9745 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:29.720618  9745 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:29.722031  9745 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.723055  9776 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:29.723825  9776 tablet_bootstrap.cc:492] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:29.724444  9776 ts_tablet_manager.cc:1403] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.342s	user 0.019s	sys 0.026s
I20260709 13:01:29.745923  9776 raft_consensus.cc:359] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:29.747467  9776 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:29.748209  9711 raft_consensus.cc:993] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:29.751140  9711 raft_consensus.cc:1081] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:29.751220  9776 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:29.753361  8726 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:29.748607  9754 raft_consensus.cc:993] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:29.753921  9776 ts_tablet_manager.cc:1434] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.002s
I20260709 13:01:29.754312  9754 raft_consensus.cc:1081] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:29.753929  8726 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:29.756052  8726 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.756279  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-9a389d19cf2c47d6b5873139d8cfad40 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:29.756708  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-9a389d19cf2c47d6b5873139d8cfad40 on tablet 9a389d19cf2c47d6b5873139d8cfad40 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:29.757762  8650 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:29.758535  8650 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:29.758924  8650 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.765008  8726 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.768797  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:29.769593  8578 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:29.769971  8578 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:29.770226  8726 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:29.770864  8578 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.772109  8578 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.777359  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:29.777971  8655 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:29.784837  8655 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:29.788200  8482 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:29.900753  8578 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:29.789944  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:01:29.944571  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 154709us
I20260709 13:01:29.946120  9728 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:29.905622  8650 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:29.910465  9791 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.144s	user 0.009s	sys 0.015s
I20260709 13:01:29.958221  8650 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:29.969655  9728 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:29.970788  9728 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:29.976271  9780 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:29.994019  9780 tablet_bootstrap.cc:492] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:30.017649  9780 ts_tablet_manager.cc:1403] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.352s	user 0.024s	sys 0.004s
I20260709 13:01:30.054690  9754 raft_consensus.cc:993] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:30.055275  9754 raft_consensus.cc:1081] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:30.070848  8651 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:30.072582  9783 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:30.075593  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:30.076323  8582 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:30.078809  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:30.079438  8733 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:30.079723  8733 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:30.080418  8733 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:30.086428  8650 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:30.089895  8582 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:30.095266  9780 raft_consensus.cc:359] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:30.096217  9780 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:30.102047  8580 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:30.102447  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:30.104391  9704 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:30.104869  9704 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:30.087249  8650 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:30.108671  8650 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:30.126031  8728 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:30.134398  8650 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:30.143833  8733 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:30.096899  9780 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:30.166941  8650 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:30.120517  9745 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 13:01:30.181016  8733 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
W20260709 13:01:30.204066  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65153.7us
I20260709 13:01:30.205763  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:30.206632  8582 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:30.214857  8582 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:30.182760  9704 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:30.221200  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:30.221498  9780 ts_tablet_manager.cc:1434] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.203s	user 0.010s	sys 0.000s
I20260709 13:01:30.222031  9745 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:30.222555  9745 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:30.223443  9745 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:30.236279  9803 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:30.254407  8728 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:30.256502  9783 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:30.262115  9787 tablet_copy_client.cc:806] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:30.262917  9787 tablet_copy_client.cc:670] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:30.280172  9330 raft_consensus.cc:1217] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:30.266358  8482 catalog_manager.cc:5697] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260709 13:01:30.287917  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 104195us
I20260709 13:01:30.292438  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:30.293081  8733 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:30.293437  8733 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:30.295849  8733 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:30.299077  8733 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:30.304059  8658 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:30.319113  8733 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:30.347280  8669 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-c3d38a98ca324ddbbb3ff25fb6ded965 received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:30.347785  8669 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-c3d38a98ca324ddbbb3ff25fb6ded965 on tablet c3d38a98ca324ddbbb3ff25fb6ded965 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:30.359846  8479 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:30.364926  8583 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:30.367007  9807 raft_consensus.cc:1064] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:30.363687  8733 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:30.398440  9752 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 13:01:30.437273  8580 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:30.438663  9806 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:30.442000  9807 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:30.488401  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 83826.9us
I20260709 13:01:30.503814  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:30.514907  8479 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:30.543004  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 124570us
I20260709 13:01:30.534425  9787 tablet_copy_client.cc:538] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:30.518262  9800 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
W20260709 13:01:30.682296  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 55414.9us
I20260709 13:01:30.770637  9787 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
W20260709 13:01:30.783944  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 239060us
W20260709 13:01:30.792979  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 235641us
W20260709 13:01:30.789029  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 245749us
I20260709 13:01:30.802527  8651 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:30.818189  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 277368us
I20260709 13:01:30.837590  8731 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:30.896188  9836 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
W20260709 13:01:31.052345  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 224912us
W20260709 13:01:31.054814  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 217581us
W20260709 13:01:31.064190  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 220672us
W20260709 13:01:31.137285  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 81610.6us
I20260709 13:01:31.157068  9839 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 13:01:31.216358  9330 raft_consensus.cc:1217] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:31.218725  9747 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:01:31.227584  8578 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:31.228881  9833 raft_consensus.cc:1064] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
W20260709 13:01:31.270365  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 126401us
I20260709 13:01:31.284929  8732 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:31.294577  9840 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:01:31.304078  8733 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:31.269536  9833 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:31.305330  9832 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20260709 13:01:31.322268  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:31.374609  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:31.418118  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 119431us
W20260709 13:01:31.430436  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:31.441473  8583 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:31.458257  8578 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:31.459474  9745 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
W20260709 13:01:31.471015  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 135319us
I20260709 13:01:31.476271  9826 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:01:31.485425  9330 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:31.495548  8731 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:31.508528  9839 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 13:01:31.530959  9330 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:31.541898  9825 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260709 13:01:31.548559  9787 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:31.522331  9807 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 13:01:31.597615  9796 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 13:01:31.589308  9787 tablet_bootstrap.cc:492] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:31.631273  9794 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 13:01:31.637209  9821 raft_consensus.cc:1064] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:31.656607  9787 ts_tablet_manager.cc:1403] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.886s	user 0.036s	sys 0.023s
W20260709 13:01:31.802474  9852 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.085s	user 0.002s	sys 0.000s
W20260709 13:01:31.836706  9863 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 13:01:31.875584  9787 raft_consensus.cc:359] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:31.879914  9787 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
W20260709 13:01:31.909338  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56773.4us
W20260709 13:01:31.932271  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 125178us
W20260709 13:01:31.940090  9866 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 13:01:31.939002  9787 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:32.016669  9872 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 13:01:32.031885  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 121800us
I20260709 13:01:31.971274  9821 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:32.058911  9787 ts_tablet_manager.cc:1434] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.401s	user 0.006s	sys 0.003s
W20260709 13:01:32.067654  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 71370.1us
W20260709 13:01:32.068298  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92517.2us
W20260709 13:01:32.094915  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110321us
W20260709 13:01:32.147017  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 114275us
I20260709 13:01:32.148546  8668 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-04399390cf414886a3453981dc1f74aa received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:32.149041  8668 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-04399390cf414886a3453981dc1f74aa on tablet 04399390cf414886a3453981dc1f74aa with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:32.149327  9826 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.168850  8582 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.171379  9328 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.208654  9329 raft_consensus.cc:1217] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:32.230835  8652 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:32.266940  8577 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:32.268623  9329 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:32.277761  8580 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:32.278434  8580 raft_consensus.cc:606] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:32.291119  9821 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:32.299731  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 165130us
W20260709 13:01:32.307546  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 163332us
W20260709 13:01:32.316252  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 170294us
I20260709 13:01:32.340019  8726 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:32.348646  9804 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:32.336625  9825 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 13:01:32.382320  9825 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:01:32.403254  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 69083.5us
W20260709 13:01:32.405290  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66758.2us
I20260709 13:01:32.355484  8481 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:32.416170  9885 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 13:01:32.444262  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 125488us
I20260709 13:01:32.455730  9793 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.523476  8730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "266236cfcd824409b757505321b36d81"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:32.525350  8730 raft_consensus.cc:606] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:32.552955  9329 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.586794  8730 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.638370  8733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:32.639228  8733 raft_consensus.cc:606] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:32.614310  8731 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.631999  9877 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:32.721647  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51700.7us
I20260709 13:01:32.725509  9794 raft_consensus.cc:993] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:32.726809  9882 raft_consensus.cc:1081] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
W20260709 13:01:32.765252  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65705.5us
I20260709 13:01:32.799787  8729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:32.800624  8729 raft_consensus.cc:606] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:32.780016  8733 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.825553  9328 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:32.867010  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 43846.3us
I20260709 13:01:32.870132  9749 raft_consensus.cc:993] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:32.870594  9728 raft_consensus.cc:993] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:32.870704  9749 raft_consensus.cc:1081] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:32.871191  9728 raft_consensus.cc:1081] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:32.874347  8659 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.908164  8729 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:32.926718  8729 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:32.935431  9826 raft_consensus.cc:1064] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:32.938884  9826 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:32.933089  8729 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.955653  8729 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:32.960193  8581 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:32.964372  8729 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:33.010758  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:33.011862  8652 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.049552  9329 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:33.050176  9329 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:33.050551  9329 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.061138  9329 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.080816  8652 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:33.074584  9328 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:33.094190  9328 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:33.094650  9328 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.100119  9329 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:33.103369  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:33.104055  8580 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.123711  9891 raft_consensus.cc:993] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
W20260709 13:01:33.138710  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61378us
I20260709 13:01:33.137979  9329 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:33.147507  8657 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.134917  9749 raft_consensus.cc:1081] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:33.159829  8657 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:33.147163  9826 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:33.160657  8657 raft_consensus.cc:606] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:33.181559  9709 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:01:33.183629  9882 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.184056  9882 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:33.186334  9882 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.186933  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:33.187668  9882 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.198334  8580 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:33.198982  8581 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:33.199635  8581 raft_consensus.cc:606] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:33.210623  8733 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.171427  8478 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.227524  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:33.228664  9854 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:33.229184  9854 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:33.230495  8478 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.243707  8582 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:33.230326  9854 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.247090  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:33.247866  8730 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.248267  8730 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:33.204164  9328 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.269317  8725 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:33.274785  9875 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 13:01:33.277668  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:33.280517  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:33.277536  9330 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "266236cfcd824409b757505321b36d81"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:33.281270  9330 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:33.281703  9330 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.249204  8730 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.284036  8730 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.288123  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:33.320277  9330 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.326968  9329 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.328485  9831 raft_consensus.cc:993] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:33.328645  9327 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.329200  9831 raft_consensus.cc:1081] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
W20260709 13:01:33.343600  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51487us
I20260709 13:01:33.401887  9330 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:33.403715  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 60214.9us
I20260709 13:01:33.408003  9704 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260709 13:01:33.409495  9328 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:33.441486  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:33.443012  8727 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:33.443320  9815 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:33.443789  8727 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:33.444260  8727 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.455287  9909 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 13:01:33.484800  9834 raft_consensus.cc:993] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:33.499411  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:33.502784  9815 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:01:33.505563  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45027.4us
I20260709 13:01:33.506073  9834 raft_consensus.cc:1081] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:33.489236  8730 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
W20260709 13:01:33.524187  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 96304.3us
I20260709 13:01:33.532738  9329 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.544399  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:33.545293  8724 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.545977  8724 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:33.546957  8724 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.561947  9891 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:33.505697  9815 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.573035  8724 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.576766  8729 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.577332  8729 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:33.578333  8729 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.579872  8729 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.580323  8727 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.582417  8727 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:01:33.625007  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 117387us
I20260709 13:01:33.633925  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:33.634601  8583 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.634874  8583 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:33.649647  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:33.650316  8581 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.654233  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:33.654912  8576 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.649159  8728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40420
I20260709 13:01:33.658315  8728 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:33.658686  8728 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.666743  8729 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:33.678141  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:33.678825  8656 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:33.679504  9908 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 13:01:33.676330  8583 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.681449  8583 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.682119  8726 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.686946  8576 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:33.690578  8583 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:33.695873  8724 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:33.698083  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:33.699131  9920 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:33.699795  9920 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:33.693933  8696 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:33.705950  8651 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:33.709352  9908 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:33.709926  9908 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:33.710825  9908 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.711810  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:01:33.712769  9815 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:33.713238  9815 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:33.713788  9889 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:33.714093  9815 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.721477  8728 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.723119  8727 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.724853  8728 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:33.726054  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:33.726627  9914 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.731227  8581 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:33.734496  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:33.701013  9920 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.738750  8580 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.764755  8652 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.769152  8576 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.773602  8652 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:33.788540  8576 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:33.793499  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:33.787819  8652 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.794468  9926 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:33.797303  8652 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.797556  9926 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:33.805079  8652 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:33.798695  9926 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:33.784135  8656 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:33.846850  8581 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:33.853260  8576 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.857486  8655 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.862110  8651 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:33.880337  9924 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 13:01:33.884409  9328 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:33.891853  9917 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:33.891010  9328 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:33.905244  8733 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.909273  8727 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:33.922657  8479 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.942123  9875 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.965909  9922 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 13:01:33.969807  8576 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:33.957644  9876 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 13:01:33.983537  9915 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 13:01:33.991681  8481 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:01:33.995128  9329 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:33.995128  9330 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:34.002377  9330 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.009997  8482 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.014076  8727 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:34.010838  9704 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 13:01:34.011842  8482 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.019649  8482 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.021379  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c3d38a98ca324ddbbb3ff25fb6ded965 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:34.028388  9709 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:34.080395  8733 raft_consensus.cc:2955] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.089371  8576 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.092483  9709 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.111658  8478 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.117893  9328 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.124315  8580 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.103920  9815 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:01:34.122665  9917 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.129381  8478 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.138629  8656 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:34.149535  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9a389d19cf2c47d6b5873139d8cfad40 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:34.157951  8733 raft_consensus.cc:2955] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.170662  8580 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:34.173043  9704 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:34.177933  8481 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.183765  8580 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.201416  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet cde36bf9fcc5409e8a00b747fbfbfb06 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:34.199785  9704 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.219267  8580 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.222393  9815 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:01:34.284621  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 56984.8us
I20260709 13:01:34.296101  9328 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:34.303973  8733 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.306474  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 04399390cf414886a3453981dc1f74aa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:34.226454  8727 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.315076  9875 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:34.315516  8583 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.318173  9815 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:34.319063  9921 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:34.320004  9920 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:34.323009  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:34.335628  8656 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:34.348368  8478 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.352190  9328 raft_consensus.cc:2955] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.353608  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet c3d38a98ca324ddbbb3ff25fb6ded965 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:34.364109  9326 raft_consensus.cc:2955] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:34.380141  8468 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 04399390cf414886a3453981dc1f74aa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20260709 13:01:34.373690  8733 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.384850  9951 tablet_replica.cc:333] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:34.386333  9891 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 13:01:34.402041  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a389d19cf2c47d6b5873139d8cfad40 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:34.403998  9921 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:34.420424  8583 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:34.421556  8581 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:34.421272  8479 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.423062  8479 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:34.565815  9749 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:34.568614  9924 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260709 13:01:34.572474  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 121671us
I20260709 13:01:34.579397  9956 tablet_replica.cc:333] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:34.580358  8655 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:34.593120  9951 raft_consensus.cc:2243] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:34.616950  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet cde36bf9fcc5409e8a00b747fbfbfb06 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:34.617763  9951 raft_consensus.cc:2272] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:01:34.659680  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58708.6us
I20260709 13:01:34.662416  9907 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:34.664009  9951 ts_tablet_manager.cc:1916] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:34.665794  8719 tablet_service.cc:1558] Processing DeleteTablet for tablet 04399390cf414886a3453981dc1f74aa with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:34.696355  9965 tablet_replica.cc:333] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
W20260709 13:01:34.722800  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 134523us
I20260709 13:01:34.745229  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:34.772161  9951 ts_tablet_manager.cc:1929] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:34.772939  9951 log.cc:1199] T c3d38a98ca324ddbbb3ff25fb6ded965 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/c3d38a98ca324ddbbb3ff25fb6ded965
I20260709 13:01:34.775101  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet c3d38a98ca324ddbbb3ff25fb6ded965 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:34.812100  9956 raft_consensus.cc:2243] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:34.812994  9956 raft_consensus.cc:2272] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:34.818853  9956 ts_tablet_manager.cc:1916] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:34.826186  9965 raft_consensus.cc:2243] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:34.827821  9965 raft_consensus.cc:2272] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:34.831302  9965 ts_tablet_manager.cc:1916] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:34.904101  9956 ts_tablet_manager.cc:1929] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:34.904704  9956 log.cc:1199] T 9a389d19cf2c47d6b5873139d8cfad40 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/9a389d19cf2c47d6b5873139d8cfad40
I20260709 13:01:34.906955  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 9a389d19cf2c47d6b5873139d8cfad40 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:34.912360  9965 ts_tablet_manager.cc:1929] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:34.912932  9965 log.cc:1199] T cde36bf9fcc5409e8a00b747fbfbfb06 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/cde36bf9fcc5409e8a00b747fbfbfb06
I20260709 13:01:34.914093  8578 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:34.914745  8578 raft_consensus.cc:606] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:34.916708  9965 tablet_replica.cc:333] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:34.916652  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet cde36bf9fcc5409e8a00b747fbfbfb06 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:34.918378  8658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "04399390cf414886a3453981dc1f74aa"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:34.918988  8658 raft_consensus.cc:606] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:34.922020  8580 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:34.922668  8580 raft_consensus.cc:606] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:34.920995  9965 raft_consensus.cc:2243] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:34.925364  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:34.926025  8731 raft_consensus.cc:606] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:34.926749  9965 raft_consensus.cc:2272] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:34.930737  8733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bf63362bb9b4409299528319b7472b04"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:34.931356  9965 ts_tablet_manager.cc:1916] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:34.931476  8733 raft_consensus.cc:606] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:34.946681  9882 raft_consensus.cc:993] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:34.947235  9709 raft_consensus.cc:1081] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:34.953373  8728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:34.953975  8728 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:34.954478  8728 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.957760  8727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:34.958415  8727 raft_consensus.cc:606] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:34.960808  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:34.962252  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:34.966145  8728 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:34.967027  9965 ts_tablet_manager.cc:1929] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:34.967588  9965 log.cc:1199] T 04399390cf414886a3453981dc1f74aa P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/04399390cf414886a3453981dc1f74aa
I20260709 13:01:34.969080  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:34.969278  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 04399390cf414886a3453981dc1f74aa (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:34.969754  8580 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.970103  8580 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:34.970924  8580 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:34.972020  8580 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.976292  8728 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:34.978294  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:34.978874  8658 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.987620  8658 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:34.988955  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:34.989936  9961 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:34.990404  9961 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:34.991255  9961 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:35.008028  8580 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:35.023574  8479 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:35.140648  9749 raft_consensus.cc:993] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:35.141199  9961 raft_consensus.cc:1081] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:35.146178  9328 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bf63362bb9b4409299528319b7472b04"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:35.146764  9328 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:35.147109  9328 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.154241  9328 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.157490  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:35.158090  8658 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.159578  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:35.160233  8731 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.160524  8731 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:35.161288  8731 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.162424  8731 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.166584  8658 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:35.167889  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:35.171307  9921 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:35.199012  9959 raft_consensus.cc:993] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:35.199599  9959 raft_consensus.cc:1081] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:35.201557  9326 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "04399390cf414886a3453981dc1f74aa"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43648
I20260709 13:01:35.202239  9326 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:35.202633  9326 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.218324  9326 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.193869  9328 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:35.223240  8731 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:35.231961  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:35.232652  8580 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.232779  9326 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:35.238082  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:35.238718  8651 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.239015  8651 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:35.239826  8651 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.241185  8651 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.233915  9921 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:35.243460  9921 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.251318  8580 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:35.254218  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:35.255409  9972 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:35.259673  8651 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:35.269762  8479 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:35.277767  9972 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:35.278887  9972 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.300526  8479 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:35.348702  9749 raft_consensus.cc:993] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:35.349306  9749 raft_consensus.cc:1081] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:35.358785  8658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:35.359413  8658 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:35.359830  8658 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.363353  9961 raft_consensus.cc:993] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:35.363850  9961 raft_consensus.cc:1081] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:35.365617  9326 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:35.366173  9326 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:35.366621  9326 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.367127  8658 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:35.369418  8658 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:35.371955  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:35.376019  8578 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.378716  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:35.379372  8727 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:35.379745  8727 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:35.380540  8727 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:35.388674  8727 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.393155  8658 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:35.397710  9922 raft_consensus.cc:993] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:35.398273  9922 raft_consensus.cc:1081] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:35.400157  9961 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:35.402213  8731 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:35.416150  8731 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:35.416532  8731 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.403330  8578 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:01:35.403483  8727 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:01:35.418069  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:35.402637  9326 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.424718  9963 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:35.425292  9963 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:35.426441  9963 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:35.443869  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:35.444509  8657 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.445117  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:35.445739  8728 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.446028  8728 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:35.446828  8728 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.447892  8728 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.448608  9326 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:35.459631  8731 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.460035  8576 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:35.461937  9961 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:35.463279  8657 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:35.467397  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:35.468567  9972 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:35.469146  9972 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:35.470207  9972 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.465442  8728 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:35.508425  8731 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:35.510612  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:35.511469  8578 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.511873  8578 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:35.514150  8578 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.515900  8578 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.517869  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:35.518699  9326 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.536221  8578 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:35.537787  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:35.532269  8479 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:35.543213  9961 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:35.548632  9961 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:35.543995  8480 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:35.552345  9326 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:35.549635  9961 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:35.593413  8480 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:35.657528  8657 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:35.659288  9972 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:36.373319  8651 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:36.018395  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:36.416131  9993 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:36.419019  8578 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:01:36.431836  8547 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.007s	sys 0.006s
W20260709 13:01:36.432286  8547 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.007s	sys 0.006s
I20260709 13:01:36.482345  8656 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:36.484197  9974 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:36.518738  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 88113.5us
I20260709 13:01:36.531049  9995 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:01:36.586598  8583 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:01:36.637323  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 90744.9us
I20260709 13:01:36.697024  9997 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:36.700969  9997 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:36.703333  9997 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:36.704769 10007 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:36.715799  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:36.743419 10007 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:36.744086 10007 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:36.772764  8582 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:36.767867 10009 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:36.774503 10008 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:36.796271 10008 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:36.796810 10011 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:36.797484 10011 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:36.799345 10008 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:36.799746 10011 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:36.807271 10007 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:36.808625  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:36.823434  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:36.846493  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:36.848724  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:36.851892 10007 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:36.815471 10009 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:36.852684  8482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 105) took 1069 ms. Trace:
I20260709 13:01:36.854365 10009 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:36.859287  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:36.870563  8733 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:36.853154  8482 rpcz_store.cc:276] 0709 13:01:35.783591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.783749 (+   158us) service_pool.cc:224] Handling call
0709 13:01:36.852594 (+1068845us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":610825,"reported_tablets":1}
W20260709 13:01:36.893553  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 69936.7us
I20260709 13:01:36.895845  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:36.897990  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:36.926164  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:36.954804  9987 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:36.981884  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50619.9us
W20260709 13:01:37.004878  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64214.8us
I20260709 13:01:37.008214  8727 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:37.010782  8728 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:01:37.010615  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60329.4us
I20260709 13:01:37.022446  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:37.048961  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 65497.7us
I20260709 13:01:37.068100  9997 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:37.068696  9997 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.073686  9326 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:37.082513  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.084492  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.086338  9709 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:37.092365  9921 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:37.094303 10025 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:37.086699  9986 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.102363  9997 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:37.117614  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 142875us
I20260709 13:01:37.118810  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:37.124876  8733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "081470ef13974fe497f88859ada43ffa"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:37.125700  8733 raft_consensus.cc:606] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:37.128539  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:37.131106  9709 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:37.110986  9921 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 13:01:37.131713  9709 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.149948  8651 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:37.150547  8651 raft_consensus.cc:606] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:37.151643  9709 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:37.124832 10009 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 13:01:37.161217  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:37.161554  8576 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:37.162147  8576 raft_consensus.cc:606] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:37.165776  8576 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:37.166344  8576 raft_consensus.cc:606] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:37.167536  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:01:37.169739  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:37.176687  8583 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:37.238535  8727 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:37.239912  9882 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:37.243374  8548 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9b3791a4a534a598bc36d62992d27be. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:37.246615  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.256354  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:37.262034  8651 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:37.264285 10019 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:37.266670 10019 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.268313 10023 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:37.254629  9882 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.268959 10019 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:37.268967 10023 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.272830 10023 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:37.273054  9987 raft_consensus.cc:993] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:37.274802  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:37.274816  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:37.277488  9987 raft_consensus.cc:1081] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:37.279788 10019 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:37.280464 10019 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.283917  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:37.285655 10019 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:37.287213  9709 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:37.287714  9709 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.295686  9709 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:37.297801  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:37.304733  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:37.307227  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:37.324200  9709 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:37.327193  8731 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
W20260709 13:01:37.331714  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcd20 after lost signal to thread 7484
W20260709 13:01:37.332788  8605 debug-util.cc:398] Leaking SignalData structure 0x7b0800085f20 after lost signal to thread 8516
W20260709 13:01:37.333982  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08001a8640 after lost signal to thread 8608
W20260709 13:01:37.337275  8605 debug-util.cc:398] Leaking SignalData structure 0x7b0800351f00 after lost signal to thread 8683
W20260709 13:01:37.338443  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08002f6820 after lost signal to thread 8757
W20260709 13:01:37.339630  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08001104c0 after lost signal to thread 9355
I20260709 13:01:37.341274 10023 raft_consensus.cc:993] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:37.343353 10023 raft_consensus.cc:1081] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:37.344909  8727 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:37.345468  8727 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:37.345813  8727 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.352651  8727 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.362071  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:37.362650  8651 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.392225  8651 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:37.387249  8480 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:37.410498  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:37.411166  8581 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:37.411492  8581 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:37.412302  8581 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.413259  8581 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.427595  8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:37.428295  8583 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:37.428751  8583 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.430714  8727 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:37.435479  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:37.436430 10024 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:37.436923 10024 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:37.437431  8583 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:37.439683 10024 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.444175  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:37.453274  8583 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:37.425125  8581 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:37.445785  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:37.457597  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:37.458328  9326 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.459120  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:37.461509  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:37.477531  9326 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:37.478442  9328 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:37.481900 10012 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:37.484182 10012 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:37.484637 10012 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:37.485617 10012 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:37.487079 10012 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.490542  8696 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.492134  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.493606 10009 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:37.494933 10009 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.506232  8657 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:37.515419  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.520376 10029 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:37.521088 10025 raft_consensus.cc:993] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:37.530195 10025 raft_consensus.cc:1081] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:37.537937  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:37.539875  9709 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:37.567052  8727 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:37.583457  9709 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:37.585799  9709 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:37.598044 10023 raft_consensus.cc:993] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:37.598716 10029 raft_consensus.cc:1081] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
W20260709 13:01:37.599924  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84588.9us
I20260709 13:01:37.601119  8726 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:37.602586  8726 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:37.603374  8726 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.626166  8482 catalog_manager.cc:5697] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:37.635499 10028 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:37.655548  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.655088  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54343.8us
I20260709 13:01:37.667562  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.670468 10023 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.670925  8549 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.674191  8657 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:37.675652  8697 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03f3a12327224113b1c5bf090e280a53. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:37.675864  9999 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:37.676025  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.674176  8726 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.687448 10028 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.688678 10028 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:37.690369  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:37.696473 10023 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.699189  8726 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:37.700425  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:37.709738  8651 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.717625  8651 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:37.719110  8698 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:37.719969  9999 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:37.720588  9999 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:37.721745  9999 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.721560 10009 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:37.728806  8651 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:37.734078  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:37.751075  8482 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:37.754554  8655 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:37.791265  8578 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "081470ef13974fe497f88859ada43ffa"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:01:37.792112  8578 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:37.792518  8578 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.804857  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:37.815811  8582 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:37.816934  8582 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:37.826303  8657 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:37.837850 10050 ts_tablet_manager.cc:933] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:37.839553  8655 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:37.842612  8580 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:37.841620  8582 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.844355  9997 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:37.845841  8582 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.891494  8582 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:37.894987  8578 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.896648  8583 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:37.901974  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:37.902722  8657 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.903524 10004 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:37.911604  8657 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:37.914872  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:01:37.915751  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47986.6us
I20260709 13:01:37.918373 10023 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:37.918936 10023 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:37.919832 10023 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.933542  8576 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:37.936370  8482 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:37.953773  8578 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:37.954885  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:37.958900  9749 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:37.962297  8726 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:37.962700  8726 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:37.963500  8726 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:37.964690  8726 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.970654 10050 tablet_copy_client.cc:323] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:37.984025  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet b9b3791a4a534a598bc36d62992d27be from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:37.984632  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet b9b3791a4a534a598bc36d62992d27be from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-b9b3791a4a534a598bc36d62992d27be
I20260709 13:01:37.993330  8593 tablet_copy_source_session.cc:215] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:37.992908  8581 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:37.997985 10050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b3791a4a534a598bc36d62992d27be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:38.004164  8651 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:37.997851  8726 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:38.005894  9749 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:38.001230  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59054.1us
I20260709 13:01:38.017937  8578 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:38.022117  9749 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:38.003526  8482 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:38.080144  8655 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:38.085633  8548 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a346f8961f774e87a3c75fa7e2053cb0. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:38.086970 10029 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:38.092698  8726 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:38.105042  8655 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:38.106863  9709 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:38.123175 10050 tablet_copy_client.cc:806] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:38.124307 10050 tablet_copy_client.cc:670] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:38.106899 10028 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 13:01:38.138862  8581 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:38.141287 10058 ts_tablet_manager.cc:933] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:38.148891 10050 tablet_copy_client.cc:538] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:38.150507  8480 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:38.162227  9329 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:38.165148 10019 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:38.170909  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:38.202126  8656 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:38.225936 10050 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:38.214171 10026 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:01:38.249389 10029 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:38.265470  9999 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:38.266867 10058 tablet_copy_client.cc:323] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:38.261932  8651 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:38.282651  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 03f3a12327224113b1c5bf090e280a53 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:38.283282  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 03f3a12327224113b1c5bf090e280a53 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-03f3a12327224113b1c5bf090e280a53
I20260709 13:01:38.319875  8743 tablet_copy_source_session.cc:215] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:38.327018  8651 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:38.331707 10058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f3a12327224113b1c5bf090e280a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:38.331494 10019 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:38.339587  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110597us
I20260709 13:01:38.339200  8726 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:38.491263  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:38.527379  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 96461.8us
I20260709 13:01:38.529146 10074 ts_tablet_manager.cc:933] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:38.532079  8482 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:38.551477  8726 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:38.554415  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 90887.7us
I20260709 13:01:38.559437 10074 tablet_copy_client.cc:323] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:38.568287  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet a346f8961f774e87a3c75fa7e2053cb0 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:38.583498  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet a346f8961f774e87a3c75fa7e2053cb0 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-a346f8961f774e87a3c75fa7e2053cb0
I20260709 13:01:38.596410 10069 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 13:01:38.598196  8727 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:38.651762 10058 tablet_copy_client.cc:806] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:38.650930  8593 tablet_copy_source_session.cc:215] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:38.654034 10058 tablet_copy_client.cc:670] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:38.687206 10058 tablet_copy_client.cc:538] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:38.695187 10050 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 13:01:38.733721  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 175894us
W20260709 13:01:38.791704  8697 consensus_peers.cc:597] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d35f944458a44f6c82498c7fec026ec5. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:38.800088  8656 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:38.804770  8656 raft_consensus.cc:606] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:38.811940 10074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a346f8961f774e87a3c75fa7e2053cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:38.807431 10050 tablet_bootstrap.cc:492] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:38.817663 10050 ts_tablet_manager.cc:1403] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.611s	user 0.037s	sys 0.020s
I20260709 13:01:38.830397 10058 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:38.838973  8651 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:38.848891 10050 raft_consensus.cc:359] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:38.856880 10027 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:38.851145  8580 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:38.858376  8580 raft_consensus.cc:606] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:38.866255  8580 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:38.871471  8727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:38.872182 10027 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:38.882818  8727 raft_consensus.cc:606] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:38.886699  8582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:38.887495  8582 raft_consensus.cc:606] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:38.859861 10050 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:38.890965  8727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
W20260709 13:01:38.892235  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 118582us
I20260709 13:01:38.888886 10050 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:38.894752  9990 raft_consensus.cc:993] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:38.892480  8727 raft_consensus.cc:606] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:38.902496  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:01:38.904222  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:38.906102  9992 raft_consensus.cc:1081] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:38.916069  8582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:38.916692  8582 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:38.917282  8582 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:38.930976  8582 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.024807  8582 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:39.029742  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 99592us
I20260709 13:01:39.031160  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:01:39.033851  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 98220.8us
I20260709 13:01:39.031754  8727 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.040174  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:39.040879  8651 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:39.041164  8651 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:39.049780  8727 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:39.051278  8651 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.052325  8651 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.054454  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:39.055662 10050 ts_tablet_manager.cc:1434] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.238s	user 0.015s	sys 0.002s
I20260709 13:01:39.062883  9709 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:39.063468  9709 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:39.064406  9709 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.089311  8651 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:39.076115  8657 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:39.035082 10025 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:39.120662 10019 raft_consensus.cc:993] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:39.128283 10088 raft_consensus.cc:1081] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:39.132979  8576 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:39.158056  8727 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:39.158730  8727 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:39.159088  8727 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.166214  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-b9b3791a4a534a598bc36d62992d27be received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:39.166739  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-b9b3791a4a534a598bc36d62992d27be on tablet b9b3791a4a534a598bc36d62992d27be with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:39.169073  8480 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:39.177997 10088 raft_consensus.cc:993] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:39.178524 10088 raft_consensus.cc:1081] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:39.180999  8726 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:39.181571  8726 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:39.181936  8726 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.184984  8727 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:39.202158  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43508.1us
I20260709 13:01:39.210183  8727 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:39.213598  8479 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:39.231887  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:39.232535  8651 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.250732  8726 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.261827  8726 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:39.268162  9328 raft_consensus.cc:1217] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:39.268378  8651 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
W20260709 13:01:39.281095  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 134833us
I20260709 13:01:39.299376  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:39.300032  8582 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:39.300334  8582 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:39.306984  8582 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.310084  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:39.310806  8575 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:39.311151  8575 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:39.313261  8575 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.314822  8575 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.318845  8582 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.321085  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:39.321784  8655 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.297307 10074 tablet_copy_client.cc:806] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:39.324957 10074 tablet_copy_client.cc:670] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:39.329243  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:39.337563 10074 tablet_copy_client.cc:538] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:39.337123  8582 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:39.340658  8575 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:39.345132 10073 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:39.395066 10058 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:39.396102 10058 tablet_bootstrap.cc:492] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
W20260709 13:01:39.396921  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48414.6us
I20260709 13:01:39.397454  9749 raft_consensus.cc:993] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:39.397818 10050 ts_tablet_manager.cc:933] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:39.398638  9749 raft_consensus.cc:1081] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:39.398828  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:39.401237  8581 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:01:39.401829  8581 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:39.402145  8581 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.418526  8655 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:39.400417 10092 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:39.429427  8581 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.429253 10058 ts_tablet_manager.cc:1403] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.599s	user 0.027s	sys 0.017s
I20260709 13:01:39.417583  9999 raft_consensus.cc:993] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:39.417725 10073 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:39.443650 10073 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.444836  8581 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:39.437481 10092 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:39.448436  9749 raft_consensus.cc:1081] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:39.448732  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:39.449311  8655 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.451252  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:39.451817  8726 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:39.452088  8726 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:39.454015  8657 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:39.454535  8657 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:39.454907  8657 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.462399  8726 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.468590  8655 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:39.473877  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:39.474648 10090 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:39.474581 10058 raft_consensus.cc:359] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:39.475297 10090 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:39.446130 10092 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.487377  8657 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.489852  8726 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.491384  8657 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:39.476397 10090 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.495301  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:39.500166  8728 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:39.500487  8728 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:39.482815 10074 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:39.501947  8728 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.475754 10058 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:39.496256  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:39.519769  8575 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.522873  8728 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.527724  8655 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:39.539512  8726 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:39.500850 10050 tablet_copy_client.cc:323] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:39.548900  8728 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:39.550256  8625 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:39.551153 10010 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:39.551648 10010 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:39.552528 10010 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:39.575835  8575 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:39.539089 10028 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 13:01:39.584056  8480 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:39.506472 10058 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:39.578567  8479 catalog_manager.cc:5697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:39.588128  8482 catalog_manager.cc:5697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:39.599891  8480 catalog_manager.cc:5697] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:39.605626  8743 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet d35f944458a44f6c82498c7fec026ec5 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:39.606205  8743 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet d35f944458a44f6c82498c7fec026ec5 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-d35f944458a44f6c82498c7fec026ec5
I20260709 13:01:39.622911 10058 ts_tablet_manager.cc:1434] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.176s	user 0.009s	sys 0.000s
I20260709 13:01:39.625015  8743 tablet_copy_source_session.cc:215] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:39.626399  8742 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-03f3a12327224113b1c5bf090e280a53 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:39.626857  8742 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-03f3a12327224113b1c5bf090e280a53 on tablet 03f3a12327224113b1c5bf090e280a53 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:39.638682 10050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35f944458a44f6c82498c7fec026ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:39.681363  8728 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:39.691231 10089 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:39.745654  9328 raft_consensus.cc:1217] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 13:01:39.756085 10104 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:01:39.783067 10074 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:39.783905 10074 tablet_bootstrap.cc:492] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:39.784503 10074 ts_tablet_manager.cc:1403] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.304s	user 0.020s	sys 0.006s
I20260709 13:01:39.833158 10050 tablet_copy_client.cc:806] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:39.838651 10050 tablet_copy_client.cc:670] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:39.846076 10050 tablet_copy_client.cc:538] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:39.837929 10074 raft_consensus.cc:359] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:39.847916 10074 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:39.848627 10074 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:39.851861 10069 raft_consensus.cc:1064] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:39.861912 10074 ts_tablet_manager.cc:1434] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.077s	user 0.007s	sys 0.004s
I20260709 13:01:39.864447  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-a346f8961f774e87a3c75fa7e2053cb0 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:39.864897  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-a346f8961f774e87a3c75fa7e2053cb0 on tablet a346f8961f774e87a3c75fa7e2053cb0 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:39.866824  9326 raft_consensus.cc:1217] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:39.854418 10069 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:39.889003 10050 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:39.905237  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:39.911808  9326 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:39.912621  8657 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:39.913504  9882 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:39.914618 10089 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:39.916131  8727 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:39.933928 10088 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:39.970193  8657 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:39.973030  9999 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:39.984817  8726 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:39.989925 10052 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:39.997212  8580 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:39.999030 10073 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:40.002990 10050 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:40.004509 10050 tablet_bootstrap.cc:492] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:40.011422  8657 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:40.017063  9999 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:40.005219 10050 ts_tablet_manager.cc:1403] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.116s	user 0.020s	sys 0.011s
I20260709 13:01:40.026804 10050 raft_consensus.cc:359] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:40.027796 10050 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:40.028487 10050 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:40.029181  8728 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:40.031354 10089 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:40.036303  8575 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:40.040445  9999 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:40.048333 10050 ts_tablet_manager.cc:1434] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.025s	user 0.003s	sys 0.004s
I20260709 13:01:40.057698  8742 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-d35f944458a44f6c82498c7fec026ec5 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:40.058177  8742 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-d35f944458a44f6c82498c7fec026ec5 on tablet d35f944458a44f6c82498c7fec026ec5 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:40.110288  8582 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:40.117170 10078 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:40.148836  8651 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:40.155026  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:40.214044 10088 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.215618  9328 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.236407 10069 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:40.306656  8728 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.339918  8651 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.343286  8481 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.356341  8656 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:40.357191  8656 raft_consensus.cc:606] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:40.363469  9326 raft_consensus.cc:1217] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:40.370565  8583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:40.371204  8583 raft_consensus.cc:606] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:40.372912 10025 raft_consensus.cc:1064] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:40.379376  8583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:40.380097  8583 raft_consensus.cc:606] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:40.382978  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:40.383730  8728 raft_consensus.cc:606] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:40.386868  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:40.387537  8728 raft_consensus.cc:606] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:40.391479  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:01:40.393548  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:40.380825 10025 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:40.416455 10078 raft_consensus.cc:993] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:40.417176 10078 raft_consensus.cc:1081] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:40.422904  8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:40.423614  8583 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:40.424021  8583 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.437652  8583 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:40.439954  8583 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:40.441299  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:40.442312  8728 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.466167  9882 raft_consensus.cc:1064] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:40.470708  8580 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:40.469098  9882 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:40.481760  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:40.482503  8656 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:40.482918  8656 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:40.483734  8656 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:40.484592  9326 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:40.487183 10124 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:40.485062  8656 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.498970 10088 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.523473  8656 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:40.528968  8726 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:40.532950 10089 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:40.543497 10090 raft_consensus.cc:993] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:40.543962 10090 raft_consensus.cc:1081] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:40.544893  9330 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:40.544844  9326 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.548453  8726 raft_consensus.cc:2955] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.563915 10025 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:40.571444  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:40.572513 10089 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:40.573020 10089 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:40.573499  8728 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:40.574762 10089 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:40.572173  8580 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:40.588423  9999 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:40.592760  8656 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:40.603057  8656 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38088
I20260709 13:01:40.603493  8656 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:40.603721  8656 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.604950  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b9b3791a4a534a598bc36d62992d27be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:40.606637  8479 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.609696  9326 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:40.611141  9997 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:40.613771  8728 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:40.618185  9997 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:40.629025  8655 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:40.638424  8656 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:40.642686  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:40.643344  9326 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.645150  8656 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:40.654567  9326 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:01:40.625684 10025 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 13:01:40.696979  8623 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:40.700094 10052 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:40.700702 10052 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:40.701661 10052 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:40.719033 10088 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:40.720911  8482 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 3, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:40.744549  9997 raft_consensus.cc:993] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:40.745049  9997 raft_consensus.cc:1081] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:40.748023  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet b9b3791a4a534a598bc36d62992d27be with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:40.753144  9328 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.774114  8726 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:40.774756  8726 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:40.775151  8726 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.748513  9999 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.785564  8726 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:40.796025  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:40.796623  8656 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:40.811910  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82747.5us
I20260709 13:01:40.813073  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:40.813671  8580 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:40.813964  8580 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:40.814798  8580 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:40.815543  8656 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:40.815835  8580 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.816846  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:40.817677  8726 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:40.824087  8580 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:01:40.827881  8651 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:40.849063  8482 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.820923 10024 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:40.882521  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:40.883268  8575 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:40.883625  8575 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:40.882624 10069 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.884429  8575 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:40.885802  8575 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.885866  8728 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.915076  8575 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:40.927167  8480 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:40.911866 10009 raft_consensus.cc:993] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:40.932533  8656 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:40.977367 10024 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:40.982492 10024 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:41.000945 10143 tablet_replica.cc:333] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:41.001556 10149 raft_consensus.cc:1081] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:41.008157 10146 raft_consensus.cc:1064] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:41.013362  8480 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:41.022012  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 192737us
I20260709 13:01:41.002105 10143 raft_consensus.cc:2243] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:41.033318 10143 raft_consensus.cc:2272] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:41.027280  8580 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.031229 10146 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:41.064141  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60701.8us
I20260709 13:01:41.066697  8656 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:41.067291  8656 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:41.067691  8656 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.067510  8655 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.083957  8656 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:41.092250  9999 raft_consensus.cc:993] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:41.098847  8656 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:41.100193  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 107952us
I20260709 13:01:41.104847  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:41.105461  8731 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:41.105737  8731 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:41.108161  8731 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:41.109249  8731 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.092808  9999 raft_consensus.cc:1081] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:41.111749 10143 ts_tablet_manager.cc:1916] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:41.123735  8482 catalog_manager.cc:5697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:41.128109  8731 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:41.137283  8582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:01:41.136546  9328 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.137850  8582 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:41.138221  8582 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.168192  9330 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:01:41.169963  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 73949.5us
I20260709 13:01:41.173645  8582 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:41.176682  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:41.177285  8581 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.177723  8728 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:41.178138  8582 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:41.199743  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:41.200569  8728 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:41.200989  8728 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:41.202014  8728 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:41.203347  8728 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.195736  8581 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:41.221487 10139 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:41.224504  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:41.225975 10160 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:41.226887 10160 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:41.227574  8728 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:41.229287  8655 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:41.231133  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:01:41.248802  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62325.9us
W20260709 13:01:41.249862 10153 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:01:41.251657 10028 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:41.212448 10052 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 13:01:41.259665  8578 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:41.275444  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:41.271095 10028 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:41.276165  8655 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.276921 10028 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:41.286072  8655 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:41.287743  8657 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:41.290876 10149 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:41.268754 10052 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20260709 13:01:41.319074  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46848.6us
I20260709 13:01:41.324931  8582 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:41.329427  8576 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:41.317555 10160 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:41.332398 10147 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 13:01:41.355963  9326 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:41.357980 10151 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:41.608072 10147 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.621430  9328 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:41.635092  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.641209  8655 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:41.640062  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 278180us
I20260709 13:01:41.670037  8578 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:41.700556 10142 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:41.452986  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:41.719149  8479 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:41.727255  8578 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:41.727857  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 75616.6us
I20260709 13:01:41.735587  9328 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:41.738546 10139 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:41.747766 10024 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:41.749038 10142 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.755549  9329 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.779346  8482 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.796133  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a346f8961f774e87a3c75fa7e2053cb0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 13:01:41.812608  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52235.8us
I20260709 13:01:41.813993 10148 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:41.817298 10143 ts_tablet_manager.cc:1929] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:41.817828 10143 log.cc:1199] T b9b3791a4a534a598bc36d62992d27be P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/b9b3791a4a534a598bc36d62992d27be
I20260709 13:01:41.818939 10143 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38080 (request call id 74) took 1141 ms. Trace:
I20260709 13:01:41.819294 10143 rpcz_store.cc:276] 0709 13:01:40.677160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.747844 (+ 70684us) service_pool.cc:224] Handling call
0709 13:01:40.990757 (+242913us) ts_tablet_manager.cc:1151] Deleting tablet b9b3791a4a534a598bc36d62992d27be
0709 13:01:40.991775 (+  1018us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 13:01:41.531504 (+539729us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:41.817150 (+285646us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:41.818858 (+  1708us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":222900,"thread_start_us":89820,"threads_started":1}
W20260709 13:01:41.823733  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 77837.5us
W20260709 13:01:41.827838  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 63825.7us
I20260709 13:01:41.828714  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet b9b3791a4a534a598bc36d62992d27be (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:41.830121  8656 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.835064  8576 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:41.838037 10052 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:41.850265  8480 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:41.852021  8480 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:41.859439  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 116114us
I20260709 13:01:41.883857  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet a346f8961f774e87a3c75fa7e2053cb0 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:41.906788  8728 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:41.915081  9329 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:41.916754  9999 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 13:01:41.921350  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.922626  8728 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:41.929859  9326 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:41.931525  9999 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:41.939213 10024 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.942348 10185 tablet_replica.cc:333] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:41.942786 10148 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.953644  9329 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.962976 10185 raft_consensus.cc:2243] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:41.973851  8726 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:41.976292 10185 raft_consensus.cc:2272] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:41.985669  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 03f3a12327224113b1c5bf090e280a53 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:41.986799  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d35f944458a44f6c82498c7fec026ec5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:41.985653  9326 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:41.993072  8650 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:42.003106  8657 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:42.022956 10185 ts_tablet_manager.cc:1916] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:42.028064  8479 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:42.038640  8578 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:42.054454 10151 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:42.039989  9997 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 13:01:42.098395  8657 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:42.088778  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:42.137580  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 88330.2us
W20260709 13:01:42.147264  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66735.2us
I20260709 13:01:42.118921 10009 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 13:01:42.163321 10185 ts_tablet_manager.cc:1929] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:42.163856 10185 log.cc:1199] T a346f8961f774e87a3c75fa7e2053cb0 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/a346f8961f774e87a3c75fa7e2053cb0
I20260709 13:01:42.154672  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 03f3a12327224113b1c5bf090e280a53 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:42.170186  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet a346f8961f774e87a3c75fa7e2053cb0 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:42.171386 10143 tablet_replica.cc:333] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:42.172562 10143 raft_consensus.cc:2243] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:42.175330  8479 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:42.173311 10143 raft_consensus.cc:2272] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:42.241045  8576 raft_consensus.cc:2955] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:42.251771  9882 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:42.255205 10195 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:42.263024  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:42.263895  8726 raft_consensus.cc:606] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:42.271178 10143 ts_tablet_manager.cc:1916] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:42.287659  8726 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:42.289211  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet d35f944458a44f6c82498c7fec026ec5 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:42.290750  8655 raft_consensus.cc:2955] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:42.323975  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:42.324673  8728 raft_consensus.cc:606] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:42.336113  8578 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:42.336743  8578 raft_consensus.cc:606] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:42.339721  8581 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:42.340348  8581 raft_consensus.cc:606] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:42.344794  8657 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:42.345357  8657 raft_consensus.cc:606] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
W20260709 13:01:42.349228  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50053.1us
I20260709 13:01:42.351476 10199 tablet_replica.cc:333] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:42.353022 10199 raft_consensus.cc:2243] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:42.353714 10199 pending_rounds.cc:70] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:01:42.354635  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:01:42.354008 10199 pending_rounds.cc:77] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7305634205817053184 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d35f944458a44f6c82498c7fec026ec5" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } } }
I20260709 13:01:42.355993 10199 raft_consensus.cc:2917] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260709 13:01:42.356501 10199 raft_consensus.cc:2889] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260709 13:01:42.356885 10199 raft_consensus.cc:2272] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:42.359885  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:42.359974 10143 ts_tablet_manager.cc:1929] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:42.360489 10143 log.cc:1199] T 03f3a12327224113b1c5bf090e280a53 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/03f3a12327224113b1c5bf090e280a53
I20260709 13:01:42.384413 10199 ts_tablet_manager.cc:1916] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:42.394676  8655 raft_consensus.cc:2955] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:42.399299  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 03f3a12327224113b1c5bf090e280a53 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:42.413213 10151 raft_consensus.cc:993] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:42.413986 10151 raft_consensus.cc:1081] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:42.417611  9329 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:42.418401  9329 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:42.418876  9329 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.430238 10199 ts_tablet_manager.cc:1929] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:42.430907 10199 log.cc:1199] T d35f944458a44f6c82498c7fec026ec5 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/d35f944458a44f6c82498c7fec026ec5
I20260709 13:01:42.440726  9329 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.448144  9329 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:42.449476  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:42.454742  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:42.455248  8731 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:42.455422  8581 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.455865  8731 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:42.457094  8731 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.458613  8731 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.445077  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet d35f944458a44f6c82498c7fec026ec5 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:42.469385  8581 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:01:42.473773 10195 raft_consensus.cc:993] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:42.491546 10195 raft_consensus.cc:1081] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:42.487846  8731 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:01:42.494494  8728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:42.495151  8728 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:42.495553  8728 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.496325  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:42.497795 10164 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:42.498374 10164 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:42.499282 10164 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.508005  8728 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.513603  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:42.514426  8581 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:42.514855  8581 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:42.515261  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:42.515938  9329 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.516016  8728 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:42.517480  8581 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.519141  8581 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.523236  9329 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:42.524641  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:01:42.525938 10147 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:42.526902 10147 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:42.527992 10147 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:42.532266  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 134979us
I20260709 13:01:42.533553 10203 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:42.544399  8480 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:42.562790  8581 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:42.575405  8480 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:42.634753 10203 raft_consensus.cc:993] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:42.635316 10203 raft_consensus.cc:1081] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:42.637774  8581 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:42.638377  8581 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:42.638803  8581 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.642321 10200 raft_consensus.cc:993] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:42.642861 10200 raft_consensus.cc:1081] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:42.644480  9329 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43640
I20260709 13:01:42.644989  9329 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:42.645310  9329 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.653466  8581 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.717489  9329 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.719555  9329 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:42.721101  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:42.721755  8578 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:42.722038  8578 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:42.722687  8578 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.723518  8578 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.724272  8581 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:42.734601  8578 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:42.735796  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:42.736390  9329 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.758178  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:42.758874  8656 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:42.759227  8656 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:42.760156  8656 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.761880  8656 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.758983  9329 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:42.775883  8656 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:42.772902  8548 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:42.782397  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:42.783082  8655 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.800217  8655 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:42.785473  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:42.794275 10142 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:42.813475 10142 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:42.815711 10142 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.811749 10164 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:42.819271 10164 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:42.820230 10164 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:42.835803 10151 raft_consensus.cc:993] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:42.836427 10024 raft_consensus.cc:1081] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:42.909747  8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:01:42.911554  8583 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:42.895745  8479 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:42.912349  8583 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.944931  9326 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:01:42.964762  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44839.8us
I20260709 13:01:42.984220 10159 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:01:43.044253  8728 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:43.046123 10214 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:43.061734  8581 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:43.093602  8583 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:43.094949 10159 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:43.099547  8583 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:43.102278  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:43.103263  8727 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:43.103653  8727 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:43.104761  8727 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:43.106559  8727 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.082913  8479 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:43.116211  8582 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:43.120718  8727 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:43.126389 10192 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:43.136281  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:43.140718  9882 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:43.141357  9882 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:43.142488  9882 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:43.167359  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:43.168324  8655 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.251520  8656 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:43.264532 10142 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:43.273767  8655 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:43.285276  8582 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:43.288066 10214 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:43.333679  8482 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:43.361449  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:43.379101  8651 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:43.380646 10214 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:43.405975  9326 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:43.413920 10089 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 13:01:43.625214  8727 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:43.627763 10089 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:43.638164  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:43.655058  8655 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:43.659200 10194 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:43.793869  8651 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:43.823005  8727 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:43.824877  8582 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:43.827854 10160 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:43.828912 10204 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:43.834887  8623 consensus_peers.cc:597] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4f31f7ca2bf4416bec3531998325cc4. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:43.837237  8655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:43.837951  8655 raft_consensus.cc:606] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:43.856303  8581 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:43.856961  8581 raft_consensus.cc:606] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:43.858666  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:01:43.860445  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:43.883141 10160 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:43.889111  8582 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:43.913893  8731 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:43.930194  8479 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:43.932282  8655 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:43.965806  8623 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f9dd5b88e0647f284658784381e1893. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:43.967790  8731 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:43.970412  8582 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:43.974771 10191 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:43.977671 10160 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:43.993775  9990 raft_consensus.cc:993] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:43.994357 10160 raft_consensus.cc:1081] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:43.996678  8578 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:43.997697  8578 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:43.998188  8578 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.011965  8578 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:44.040766  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:44.041524  9326 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.042253  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:44.042943  8651 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:44.043262  8651 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:44.044183  8651 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:44.045352  8651 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.052651  9326 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:01:44.053685 10160 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.054642  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:44.058974 10089 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:44.059602 10089 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:44.059157  8731 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.060694 10089 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:44.073129  8582 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.053651  8578 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:44.076542  8651 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:01:44.107944  8731 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:44.116201  8482 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.148444 10089 raft_consensus.cc:993] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:44.149004 10089 raft_consensus.cc:1081] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:44.152954  8728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:44.153538  8728 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:44.153887  8728 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:44.162803  8728 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:44.165185  8479 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 4, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 4 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:44.174547  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:44.175303  8582 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:44.175637  8582 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:44.176512  8582 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:44.179000  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:44.179694  8655 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:44.179168  8582 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:44.195231  8728 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:44.198892  8582 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:44.200425  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:44.201444 10024 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:44.201933 10024 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:44.202850 10024 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:44.202802  8655 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:44.208690  8651 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
W20260709 13:01:44.213454  8697 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8b894d0fc374f33a626d5fd9d8ca4e7. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:44.218204  8657 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:44.220121 10209 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:44.230123  8582 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:44.238338 10244 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:44.270576  8479 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:44.302690 10209 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.319602  8655 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.346364  8583 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.358755  8582 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:44.365729  8482 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:44.375355  8548 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94c0917e28b9498a823f74b5f664a815. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:44.381716  8728 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:44.383325  9997 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:44.387785  8655 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:44.391215  9997 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:44.410136 10142 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.414091  8731 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.433135  8657 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.449361 10252 ts_tablet_manager.cc:933] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:44.470562  8482 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:44.531770  9328 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:44.535115 10089 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:44.552333 10252 tablet_copy_client.cc:323] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:44.556383  8668 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet a4f31f7ca2bf4416bec3531998325cc4 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:44.556984  8668 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet a4f31f7ca2bf4416bec3531998325cc4 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-a4f31f7ca2bf4416bec3531998325cc4
I20260709 13:01:44.570618  8668 tablet_copy_source_session.cc:215] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:44.589525 10252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f31f7ca2bf4416bec3531998325cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:44.609777  8655 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:44.613345 10142 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:44.620765  8657 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:44.622561 10244 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:44.674459 10255 ts_tablet_manager.cc:933] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:44.694016 10252 tablet_copy_client.cc:806] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:44.694310 10260 ts_tablet_manager.cc:933] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:44.700786 10252 tablet_copy_client.cc:670] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:44.724689  8582 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:44.726523 10263 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:44.787874 10252 tablet_copy_client.cc:538] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:44.787933 10255 tablet_copy_client.cc:323] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:44.800215 10252 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:44.815522  8668 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet 9f9dd5b88e0647f284658784381e1893 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:44.822340  8668 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet 9f9dd5b88e0647f284658784381e1893 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-9f9dd5b88e0647f284658784381e1893
I20260709 13:01:44.823578 10260 tablet_copy_client.cc:323] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:44.826376  8742 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet b8b894d0fc374f33a626d5fd9d8ca4e7 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:44.828022  8742 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet b8b894d0fc374f33a626d5fd9d8ca4e7 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-b8b894d0fc374f33a626d5fd9d8ca4e7
I20260709 13:01:44.846638  8742 tablet_copy_source_session.cc:215] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:44.848698  8668 tablet_copy_source_session.cc:215] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:44.859153 10255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9dd5b88e0647f284658784381e1893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:44.861593  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:44.869901 10260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b894d0fc374f33a626d5fd9d8ca4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:44.903187 10267 ts_tablet_manager.cc:933] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:44.928025 10255 tablet_copy_client.cc:806] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:44.928843 10255 tablet_copy_client.cc:670] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:44.943949 10260 tablet_copy_client.cc:806] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:44.944591 10260 tablet_copy_client.cc:670] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:44.945340 10255 tablet_copy_client.cc:538] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:44.954051 10260 tablet_copy_client.cc:538] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:44.958583 10267 tablet_copy_client.cc:323] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:44.961141  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 94c0917e28b9498a823f74b5f664a815 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:44.962046  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 94c0917e28b9498a823f74b5f664a815 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-94c0917e28b9498a823f74b5f664a815
I20260709 13:01:44.965101 10260 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:44.965761 10255 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:44.977424  8593 tablet_copy_source_session.cc:215] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:44.983533 10267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94c0917e28b9498a823f74b5f664a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:45.020700 10267 tablet_copy_client.cc:806] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:45.021348 10267 tablet_copy_client.cc:670] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:45.034601 10267 tablet_copy_client.cc:538] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:45.049660 10267 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:45.067368 10252 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:45.070621 10252 tablet_bootstrap.cc:492] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:45.074522 10252 ts_tablet_manager.cc:1403] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.275s	user 0.021s	sys 0.011s
I20260709 13:01:45.079854 10260 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:45.087877 10260 tablet_bootstrap.cc:492] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:45.098296 10260 ts_tablet_manager.cc:1403] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.133s	user 0.023s	sys 0.011s
I20260709 13:01:45.084651 10252 raft_consensus.cc:359] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.108634 10252 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:45.121034 10252 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.124172 10252 ts_tablet_manager.cc:1434] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 13:01:45.127473 10260 raft_consensus.cc:359] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.130581 10260 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:45.131305 10260 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.133811  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:45.138768  8668 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-a4f31f7ca2bf4416bec3531998325cc4 received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:45.139339  8668 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-a4f31f7ca2bf4416bec3531998325cc4 on tablet a4f31f7ca2bf4416bec3531998325cc4 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:45.142264 10260 ts_tablet_manager.cc:1434] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.003s
I20260709 13:01:45.146193  8742 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-b8b894d0fc374f33a626d5fd9d8ca4e7 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:45.146663  8742 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-b8b894d0fc374f33a626d5fd9d8ca4e7 on tablet b8b894d0fc374f33a626d5fd9d8ca4e7 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:45.162587 10267 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:45.163450  9328 raft_consensus.cc:1217] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:45.165381 10267 tablet_bootstrap.cc:492] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:45.168275 10255 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:45.169654 10255 tablet_bootstrap.cc:492] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:45.170526 10255 ts_tablet_manager.cc:1403] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.205s	user 0.028s	sys 0.033s
I20260709 13:01:45.168390 10267 ts_tablet_manager.cc:1403] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.119s	user 0.038s	sys 0.013s
I20260709 13:01:45.195874 10255 raft_consensus.cc:359] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.208109 10255 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:45.208989 10255 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.200953 10267 raft_consensus.cc:359] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.230916 10267 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:45.232064 10267 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:45.235258 10267 ts_tablet_manager.cc:1434] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.053s	user 0.005s	sys 0.003s
I20260709 13:01:45.237609  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-94c0917e28b9498a823f74b5f664a815 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:45.245375  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-94c0917e28b9498a823f74b5f664a815 on tablet 94c0917e28b9498a823f74b5f664a815 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:45.235417 10255 ts_tablet_manager.cc:1434] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.064s	user 0.004s	sys 0.004s
I20260709 13:01:45.250046  8668 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-9f9dd5b88e0647f284658784381e1893 received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:45.250512  8668 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-9f9dd5b88e0647f284658784381e1893 on tablet 9f9dd5b88e0647f284658784381e1893 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:45.253621  8581 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:45.254334  8581 raft_consensus.cc:606] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:45.274065  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:45.274747  8726 raft_consensus.cc:606] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:45.280716  8655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:45.281363  8655 raft_consensus.cc:606] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:45.284071  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:01:45.285722  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:45.385584  9328 raft_consensus.cc:1217] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:45.503834  9330 raft_consensus.cc:1217] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:45.541165 10191 raft_consensus.cc:993] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:45.543597 10191 raft_consensus.cc:1081] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:45.718189  8582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:45.718750  8582 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:45.719061  8582 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:45.784123  8582 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:45.799090  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 76323.9us
I20260709 13:01:45.801986  8582 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:45.811081  9330 raft_consensus.cc:1217] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 13:01:45.849889  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 83684.8us
W20260709 13:01:45.854023  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51758.8us
W20260709 13:01:45.856724  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58948.8us
I20260709 13:01:45.928050  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:45.928928  8655 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:45.929304  8655 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:45.930217  8655 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:45.950276  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 77689.4us
W20260709 13:01:45.952446  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 128119us
I20260709 13:01:45.957439  8655 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:46.004848  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 52124.7us
I20260709 13:01:46.007593  8655 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:46.024636 10292 raft_consensus.cc:1064] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:46.027523 10292 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:46.041800  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:46.042466  8728 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:46.046434  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 201633us
I20260709 13:01:46.070941 10288 raft_consensus.cc:993] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:46.071441 10288 raft_consensus.cc:1081] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
W20260709 13:01:46.074198  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 216827us
I20260709 13:01:46.084978 10275 raft_consensus.cc:993] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:46.085587 10275 raft_consensus.cc:1081] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:46.052068  8728 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:46.112430 10299 raft_consensus.cc:1064] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:46.115535 10299 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:46.118301 10204 raft_consensus.cc:1064] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:46.121163 10204 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:46.141594  8550 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:46.142684 10293 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:46.142757  8656 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:46.143533  8656 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:46.143937 10293 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:46.145001 10293 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:46.143954  8656 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.147118  9330 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:46.161629  8727 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:46.162283  8727 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:46.162657  8727 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:46.171778  8658 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:46.173791  8727 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:46.175135  8731 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:46.178347  8727 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:46.183732  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:46.184379  9326 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:46.188804  8731 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:46.190490  8656 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:46.193236  8656 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:46.196866  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:46.197635  8731 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:46.198017  8731 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:46.203934  9326 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:46.257527 10281 raft_consensus.cc:1064] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:46.259090  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:01:46.211880  9330 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:46.217836  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:46.261300  8582 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.261613 10292 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:46.262188 10292 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:46.263033 10292 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:46.219041  8583 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:46.222293  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:46.236675 10288 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:46.238698 10142 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:46.267079 10299 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:46.247045  8656 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:46.271397  9330 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:46.271418  8579 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:46.202742 10291 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260709 13:01:46.272387  8579 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:46.273247  8579 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:46.274143 10283 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:46.276538  8726 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:46.274483  8579 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:46.205340  8731 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:46.278460  8731 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.278954  9328 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
W20260709 13:01:46.279289 10285 consensus_queue.cc:1175] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0d703ee18464475a290425976246555" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260709 13:01:46.287262  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:46.288489  8582 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:01:46.298558  8731 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:01:46.301982 10291 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:01:46.301326 10299 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 13:01:46.302318  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:46.311945 10287 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:46.312470 10287 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:46.316165 10278 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 13:01:46.313310 10287 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:46.328025  8583 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:46.328855  8579 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
W20260709 13:01:46.330930 10288 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 13:01:46.267951 10281 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:46.369415 10303 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 13:01:46.370057 10284 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 13:01:46.371186 10290 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:46.443732  8651 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:46.451586  8481 catalog_manager.cc:5697] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:46.454331  8482 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:46.451315 10283 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:46.481621  8582 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:46.483613  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 86438.3us
I20260709 13:01:46.503163 10297 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:46.515686  8726 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:46.564903  8479 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:46.622572  9326 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:46.610383 10317 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:01:46.643756  9328 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:01:46.645211  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 130458us
I20260709 13:01:46.649083  8731 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:01:46.657648  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 116959us
I20260709 13:01:46.671373  8651 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:46.682622 10314 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.169s	user 0.004s	sys 0.000s
I20260709 13:01:46.660162  9328 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:46.705435 10284 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:46.659714 10313 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 13:01:46.737389  8480 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:46.733026 10160 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 13:01:46.754601  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:46.759845  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92932.9us
I20260709 13:01:46.849112  8580 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:01:46.884068  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56828us
I20260709 13:01:46.887131 10287 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:01:46.951632  8580 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:46.954137  8656 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:46.956606  8728 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:46.962589  8731 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:46.967939  9330 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:01:46.969791 10333 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:01:46.980399 10160 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:46.994143  8582 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:46.995621 10288 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:47.003913  9326 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.011768 10330 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:47.032009  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51704.5us
I20260709 13:01:47.053905  8731 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.064260 10330 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:47.066255 10329 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.046257 10280 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:47.167367  9326 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.174008  8580 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.175383  8578 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.229377  8482 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.231345  8482 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:47.237265 10288 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:47.244489  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:47.246783  8731 raft_consensus.cc:606] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:47.246505  8482 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:47.249316 10263 raft_consensus.cc:993] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:47.249827 10263 raft_consensus.cc:1081] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:47.253317  8582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:01:47.253898  8582 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:47.254283  8582 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.254032  8580 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:47.255007  8580 raft_consensus.cc:606] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:47.261462  8582 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:47.264000  8658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:47.284513  8658 raft_consensus.cc:606] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:47.266206  8731 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.288421  8651 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:47.304656  8582 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:47.334983  8658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:47.335587  8658 raft_consensus.cc:606] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:47.349701  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:47.354736  9330 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:47.342079  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:47.355582  8582 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:47.356894  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:01:47.359308 10280 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:47.360797  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:47.363929  8728 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:47.367399  8655 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:47.376113  9330 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:47.378465 10318 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:47.354755  8731 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:47.355779  8726 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:47.355840  8651 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.385739  8726 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:47.386603  8726 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:47.387815  8726 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.389474 10288 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.393905 10278 raft_consensus.cc:993] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:47.394625 10278 raft_consensus.cc:1081] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:47.394810  9326 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:47.401075  9326 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 13:01:47.404980  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50790.9us
I20260709 13:01:47.391892 10160 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 13:01:47.409536  8651 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:47.411914  8658 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:47.414374  8580 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:47.417171 10329 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:47.417685 10278 raft_consensus.cc:993] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:47.398236  9330 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.418179 10324 raft_consensus.cc:1081] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:47.421957  8731 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40420
I20260709 13:01:47.422596  8731 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:47.423039  8731 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.430683  8582 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:47.414758  8549 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:47.438380  8726 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:47.440914  8731 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:47.450970  8731 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:47.451227  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:47.451915  8658 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:47.452261  9330 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.453411  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:47.454008  9328 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.452216  8658 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:47.461580  8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:47.462239  8583 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:47.462587  8583 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.460498  8658 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:47.447418 10299 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:47.482604  8583 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:47.488049  9328 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:47.490219  8658 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.491205  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:47.491840  8651 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:47.492228  8651 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:47.496423  8583 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:47.496544  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:01:47.498109  8651 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:47.506954 10283 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:47.507556 10283 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:47.506168 10278 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.512588 10328 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260709 13:01:47.471575 10280 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 13:01:47.514213 10324 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.515576  8479 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.452775 10331 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:01:47.532141  8651 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.530968  8656 raft_consensus.cc:2955] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.536370  8578 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.560878  8658 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
W20260709 13:01:47.564844  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 117523us
I20260709 13:01:47.571650  8651 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:47.574106  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:47.508927 10283 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:47.580997  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 108155us
W20260709 13:01:47.611786  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 115300us
I20260709 13:01:47.576956 10330 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 13:01:47.610494 10194 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:47.636755  9328 raft_consensus.cc:2955] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.604866 10299 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:47.681691  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:47.682389  8733 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:47.697177  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 94c0917e28b9498a823f74b5f664a815 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 13:01:47.720743  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 111431us
I20260709 13:01:47.720419  8731 raft_consensus.cc:2955] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:47.726929  8733 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:47.723870 10194 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:47.769538 10299 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:47.778916 10194 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:47.784446  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a4f31f7ca2bf4416bec3531998325cc4 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 13:01:47.842749  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 160682us
W20260709 13:01:47.883905  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 95412us
W20260709 13:01:47.884104  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 89799.2us
W20260709 13:01:47.925534  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 131819us
W20260709 13:01:47.984427 10315 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 13:01:47.985947 10366 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
W20260709 13:01:48.001483  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 90146.9us
W20260709 13:01:48.042411 10374 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.054s	user 0.003s	sys 0.000s
I20260709 13:01:48.043915  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet 94c0917e28b9498a823f74b5f664a815 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:48.136142 10380 tablet_replica.cc:333] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:48.137342 10380 raft_consensus.cc:2243] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:48.138260 10380 raft_consensus.cc:2272] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:48.153088  8479 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:48.166253  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 89049.5us
I20260709 13:01:48.164386 10380 ts_tablet_manager.cc:1916] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:48.204031 10318 raft_consensus.cc:993] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:48.244445 10331 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:48.215790 10349 raft_consensus.cc:1081] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:48.267670  9328 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:48.272171  9329 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:48.298182  8650 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:48.297873 10351 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 13:01:48.326700  8659 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:48.329279  8728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:48.329957  8728 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:48.330354  8728 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:48.339197  8733 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:48.350340  8659 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:48.351778  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9f9dd5b88e0647f284658784381e1893 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:48.355119 10361 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:01:48.362619  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:48.353729  8728 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:48.365197  8733 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:48.367219  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:48.367367  8728 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:48.367839  8580 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:48.368289  8580 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:48.368970  8580 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:48.370026  8580 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:48.369028  8575 raft_consensus.cc:2955] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:48.361006  8480 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:48.388129  8480 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:48.408241 10299 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:48.408854  8728 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:48.411782 10361 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:48.412706 10288 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:48.406618  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:48.423583  8659 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:48.425177  8580 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:48.426234 10355 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:48.433305 10377 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.064s	user 0.003s	sys 0.000s
I20260709 13:01:48.438686  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:48.450492 10285 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:48.451445 10285 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:48.452814 10285 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:48.490171  8659 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:48.443032 10331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40398 (request call id 75) took 1096 ms. Trace:
I20260709 13:01:48.501206 10331 rpcz_store.cc:276] 0709 13:01:47.352288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.352843 (+   555us) service_pool.cc:224] Handling call
0709 13:01:47.384846 (+ 32003us) log.cc:844] Serialized 487 byte log entry
0709 13:01:48.438807 (+1053961us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":860,"raft.run_cpu_time_us":993,"raft.run_wall_time_us":994}
I20260709 13:01:48.503185  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b8b894d0fc374f33a626d5fd9d8ca4e7 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:48.524164 10324 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:48.554942 10324 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:48.558003 10388 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:01:48.591522 10324 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:48.592809  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:48.593501  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:48.596530  8625 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:48.592877 10380 ts_tablet_manager.cc:1929] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:48.600849 10324 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:48.599896 10380 log.cc:1199] T 94c0917e28b9498a823f74b5f664a815 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/94c0917e28b9498a823f74b5f664a815
I20260709 13:01:48.618969  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 94c0917e28b9498a823f74b5f664a815 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:48.666448  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:48.684051 10318 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:48.684659 10318 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:48.686327  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet a4f31f7ca2bf4416bec3531998325cc4 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:48.688333 10318 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:48.688966  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:48.703495  8482 catalog_manager.cc:5697] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:48.716442  8482 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:48.718937  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet 9f9dd5b88e0647f284658784381e1893 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:48.720314 10380 tablet_replica.cc:333] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:48.726508  8480 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:48.742753  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:48.746284  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:48.749516 10400 tablet_replica.cc:333] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:48.750949 10400 raft_consensus.cc:2243] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:48.751916 10400 raft_consensus.cc:2272] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:48.768005  8550 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:48.771948 10351 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:01:48.779935  8550 raft_consensus.cc:2686] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 13:01:48.807931 10400 ts_tablet_manager.cc:1916] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:48.786597 10351 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:48.831815 10380 raft_consensus.cc:2243] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:48.861914  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:48.889659  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:48.898663  8698 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:48.900087 10283 raft_consensus.cc:2706] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Received pre-election callback for term 2 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 13:01:48.902065 10351 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:48.909631  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:48.910871  8578 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:48.911429 10380 raft_consensus.cc:2272] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:48.916607 10380 ts_tablet_manager.cc:1916] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:48.917732 10351 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:01:48.920409  8479 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:48.924978  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet b8b894d0fc374f33a626d5fd9d8ca4e7 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:48.935165 10380 ts_tablet_manager.cc:1929] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:48.936642 10380 log.cc:1199] T 9f9dd5b88e0647f284658784381e1893 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/9f9dd5b88e0647f284658784381e1893
I20260709 13:01:48.938198  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 9f9dd5b88e0647f284658784381e1893 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:48.960178 10400 ts_tablet_manager.cc:1929] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:48.971385 10401 tablet_replica.cc:333] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:48.976080  8657 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:48.978477 10283 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:48.995610 10401 raft_consensus.cc:2243] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:48.997282 10401 raft_consensus.cc:2272] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:49.017165 10401 ts_tablet_manager.cc:1916] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:49.005270 10400 log.cc:1199] T a4f31f7ca2bf4416bec3531998325cc4 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/a4f31f7ca2bf4416bec3531998325cc4
I20260709 13:01:49.027596  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet a4f31f7ca2bf4416bec3531998325cc4 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:49.086489 10401 ts_tablet_manager.cc:1929] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:49.087121 10401 log.cc:1199] T b8b894d0fc374f33a626d5fd9d8ca4e7 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/b8b894d0fc374f33a626d5fd9d8ca4e7
I20260709 13:01:49.090600  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet b8b894d0fc374f33a626d5fd9d8ca4e7 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:49.209069  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:49.265457  8733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:49.266152  8733 raft_consensus.cc:606] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:49.269086  8578 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:49.269666  8578 raft_consensus.cc:606] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:49.273465  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:49.274128  8728 raft_consensus.cc:606] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:49.276909  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea2142382a5d4439915e07c70837c688"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:49.277552  8728 raft_consensus.cc:606] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:49.280612  8578 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:49.281203  8578 raft_consensus.cc:606] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:49.282296  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:01:49.284142  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:49.305022 10283 raft_consensus.cc:993] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:49.305692 10283 raft_consensus.cc:1081] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:49.308194  8655 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea2142382a5d4439915e07c70837c688"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:49.308815  8655 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:49.309190  8655 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.318698  8655 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.320988  8655 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:49.322043  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:49.322500  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:49.323084  8578 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.322755  8728 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:49.328404  8728 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:49.329187  8728 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.330404  8728 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.340600  8578 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:49.341770  8622 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:49.342661 10376 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:49.343160 10376 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:49.343986 10376 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.347366  8728 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:49.367540  8481 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:49.474855 10318 raft_consensus.cc:993] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:49.475426 10318 raft_consensus.cc:1081] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:49.478474  8733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:49.479501  8733 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:49.479873  8733 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.490387  8733 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.493049  8733 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:49.493733  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:49.494451  8652 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.495642  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:49.504191  8580 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:49.505024  8580 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:49.507262  8580 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.508381  8580 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.512771  8652 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:49.514065  8698 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:49.514232  8580 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:49.515574 10283 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:49.516070 10283 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:49.516913 10283 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.531085  8481 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:49.544164 10283 raft_consensus.cc:993] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:49.545305 10351 raft_consensus.cc:1081] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:49.548923  9326 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:01:49.549540  9326 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:49.549904  9326 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:49.557634  9326 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:49.560992  9326 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:49.561137  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:49.561554  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:49.561759  8580 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:49.562120  8728 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:49.562427  8728 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:49.563112  8728 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:49.564211  8728 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:49.567543  8580 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:01:49.571852  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:49.574263 10378 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:49.572990  8728 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:01:49.587548 10378 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:49.588822 10378 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:49.600294  8481 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:49.656315 10318 raft_consensus.cc:993] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:49.656951 10318 raft_consensus.cc:1081] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:49.659785  8652 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38088
I20260709 13:01:49.664031  8652 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:49.664455  8652 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.672575  8652 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:49.674368  8652 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:49.675364  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:49.675513  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:49.676090  8580 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:49.676146  9326 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.676551  8580 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:49.677469  8580 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:49.678942  8580 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.687899 10283 raft_consensus.cc:993] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:49.688472 10283 raft_consensus.cc:1081] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:49.690096  8580 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:49.690683  8652 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:49.691258  8652 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:49.691175  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:49.689713  9326 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:49.692576 10287 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:49.692799  8652 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.693179 10287 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:49.694252 10287 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:49.705120  8652 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.708670  8652 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:49.709043  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:49.709872  8728 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:49.710224  8728 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:49.712621  8728 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.713944  8728 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.721102  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:49.721807  8580 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:49.727157  8481 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:49.729086  8580 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:49.731192  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:49.732256 10287 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:49.732890 10287 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:49.735083 10287 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:49.735991  8728 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:49.751116  8481 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:49.904539  8580 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:49.905886 10367 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:49.955269  8580 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:49.960055 10287 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:49.964294  8727 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:49.982631 10376 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:49.995048  8652 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:49.996929 10351 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:50.024408  8575 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:50.026348 10378 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:50.180507  8728 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:50.190884 10406 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:01:50.198515  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 112152us
W20260709 13:01:50.206035  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 106375us
I20260709 13:01:50.258338  9326 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:50.261950 10287 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:50.301754  8728 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:50.302858  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:50.324440  8581 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:50.331779 10416 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 13:01:50.353169  8578 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:50.393783 10368 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:50.396556 10368 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:50.409293  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 77797.6us
W20260709 13:01:50.450423  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70362us
I20260709 13:01:50.544256  8655 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:50.555398  8575 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 13:01:50.570634  8623 consensus_peers.cc:597] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea2142382a5d4439915e07c70837c688. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:50.575701 10368 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:50.610365  8731 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:50.622532 10439 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:50.636744 10368 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.657945  8578 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.674520  8480 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:50.696213  8581 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:50.737648  8728 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:50.739306 10441 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:50.740123  8655 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 13:01:50.742650  8548 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5559db905acf4ba0b48e9e3e87f49c1c. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:50.736392  8727 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.749555 10430 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:50.767617 10430 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.770905  8728 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.808490  8727 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:50.821177  8655 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.824348  8479 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:50.838932  8581 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:50.839725  8652 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:50.840682 10433 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:50.841588 10414 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:01:50.856597  8697 consensus_peers.cc:597] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78613f64484447c487fa71abda1ec1ac. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:50.871598 10414 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.877537  8581 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.894994  8652 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.907644  8481 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:50.911752  8581 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:50.940658  8728 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 13:01:50.942029  8548 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b387b386886048c8b66e5f499f02aab2. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:50.942183 10441 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:50.947599  8652 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:50.951553 10318 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:50.961711 10350 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.971019  8655 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.977782  8727 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:50.996883  8480 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:51.013578 10460 ts_tablet_manager.cc:933] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:51.058820 10460 tablet_copy_client.cc:323] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:51.064047  8668 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet ea2142382a5d4439915e07c70837c688 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:51.064653  8668 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet ea2142382a5d4439915e07c70837c688 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-ea2142382a5d4439915e07c70837c688
I20260709 13:01:51.073127  8668 tablet_copy_source_session.cc:215] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:51.080837 10460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2142382a5d4439915e07c70837c688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:51.107306 10460 tablet_copy_client.cc:806] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:51.108074 10460 tablet_copy_client.cc:670] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:51.112061 10460 tablet_copy_client.cc:538] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:51.120608 10460 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:51.158325 10463 ts_tablet_manager.cc:933] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:51.167320 10460 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:51.168816 10460 tablet_bootstrap.cc:492] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:51.168675 10463 tablet_copy_client.cc:323] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:51.169793 10460 ts_tablet_manager.cc:1403] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.049s	user 0.026s	sys 0.007s
I20260709 13:01:51.173113  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 5559db905acf4ba0b48e9e3e87f49c1c from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:51.173794  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 5559db905acf4ba0b48e9e3e87f49c1c from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-5559db905acf4ba0b48e9e3e87f49c1c
I20260709 13:01:51.177491 10460 raft_consensus.cc:359] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.178457 10460 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:51.180580 10460 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.190166  8593 tablet_copy_source_session.cc:215] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:51.194590 10463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5559db905acf4ba0b48e9e3e87f49c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:51.195253 10460 ts_tablet_manager.cc:1434] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.001s
I20260709 13:01:51.200965  8668 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-ea2142382a5d4439915e07c70837c688 received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:51.201496  8668 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-ea2142382a5d4439915e07c70837c688 on tablet ea2142382a5d4439915e07c70837c688 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:51.230233 10463 tablet_copy_client.cc:806] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:51.231153 10463 tablet_copy_client.cc:670] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:51.236085 10463 tablet_copy_client.cc:538] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:51.248700 10463 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:51.321839 10463 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:51.323359 10463 tablet_bootstrap.cc:492] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:51.324249 10463 ts_tablet_manager.cc:1403] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.076s	user 0.046s	sys 0.019s
I20260709 13:01:51.330191 10463 raft_consensus.cc:359] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.331439 10463 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:51.332249 10463 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.336604 10463 ts_tablet_manager.cc:1434] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:01:51.338805  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-5559db905acf4ba0b48e9e3e87f49c1c received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:51.339226  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-5559db905acf4ba0b48e9e3e87f49c1c on tablet 5559db905acf4ba0b48e9e3e87f49c1c with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:51.444487 10463 ts_tablet_manager.cc:933] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:51.447142  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:51.449203 10463 tablet_copy_client.cc:323] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:01:51.451606  8742 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 78613f64484447c487fa71abda1ec1ac from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:01:51.452173  8742 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 78613f64484447c487fa71abda1ec1ac from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-78613f64484447c487fa71abda1ec1ac
I20260709 13:01:51.460788  8742 tablet_copy_source_session.cc:215] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:51.463961 10463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78613f64484447c487fa71abda1ec1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:51.485657 10463 tablet_copy_client.cc:806] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:51.486428 10463 tablet_copy_client.cc:670] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:51.490446 10463 tablet_copy_client.cc:538] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:51.509848 10463 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:51.538345  9326 raft_consensus.cc:1217] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:51.553910 10460 ts_tablet_manager.cc:933] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:51.573711  8581 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:51.574385  8581 raft_consensus.cc:606] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:51.579957  8655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:51.582212  8655 raft_consensus.cc:606] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:51.585700 10460 tablet_copy_client.cc:323] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:51.590039  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:51.590682  8728 raft_consensus.cc:606] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:51.592725  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:01:51.594383  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet b387b386886048c8b66e5f499f02aab2 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:51.594570  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:51.595008  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet b387b386886048c8b66e5f499f02aab2 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-b387b386886048c8b66e5f499f02aab2
I20260709 13:01:51.604966 10463 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:51.605901  8593 tablet_copy_source_session.cc:215] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:51.606101 10463 tablet_bootstrap.cc:492] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:51.606983 10463 ts_tablet_manager.cc:1403] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.097s	user 0.030s	sys 0.002s
I20260709 13:01:51.609673 10460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b387b386886048c8b66e5f499f02aab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:51.610237 10463 raft_consensus.cc:359] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.611191 10463 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:51.611929 10463 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.617973 10463 ts_tablet_manager.cc:1434] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20260709 13:01:51.620607  8742 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-78613f64484447c487fa71abda1ec1ac received from {username='slave'} at 127.0.0.1:42702
I20260709 13:01:51.621040  8742 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-78613f64484447c487fa71abda1ec1ac on tablet 78613f64484447c487fa71abda1ec1ac with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:51.630237 10460 tablet_copy_client.cc:806] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:51.630980 10460 tablet_copy_client.cc:670] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:51.634752 10460 tablet_copy_client.cc:538] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:51.647287 10460 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:51.736706 10466 raft_consensus.cc:993] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:51.737462 10466 raft_consensus.cc:1081] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:51.761245  9326 raft_consensus.cc:1217] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:51.800192  8578 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:51.800968  8578 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:51.801476  8578 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.874974  8578 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:51.876901  8578 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:51.878564  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:51.884012  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:51.886268  8728 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.899770 10460 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:51.900755 10460 tablet_bootstrap.cc:492] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:51.901510 10460 ts_tablet_manager.cc:1403] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.254s	user 0.062s	sys 0.071s
I20260709 13:01:51.902771 10350 raft_consensus.cc:993] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:51.903405 10350 raft_consensus.cc:1081] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:51.904639 10460 raft_consensus.cc:359] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.905102  8727 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:51.905735  8727 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:51.906059  8727 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.909976  8728 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:51.911171  8655 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:51.911481  8655 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:51.911418  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:51.912503  8655 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:51.912756  8727 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:51.915381 10423 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:51.915563  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:51.915051 10460 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:51.916767 10460 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:51.918819  8727 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:51.922251  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:51.922989  8659 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.922565 10460 ts_tablet_manager.cc:1434] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 13:01:51.937155  8659 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:01:51.938517  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:51.939327 10470 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:51.939808 10470 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:51.940682 10470 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:51.943670  8578 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:51.944068  8578 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:51.916028 10423 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:51.944739  8578 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:51.946116  8578 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.952317  8655 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.960417  9326 raft_consensus.cc:1217] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:51.964800  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-b387b386886048c8b66e5f499f02aab2 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:51.965279  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-b387b386886048c8b66e5f499f02aab2 on tablet b387b386886048c8b66e5f499f02aab2 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:51.966784  8655 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:51.945612 10423 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:51.991400  8578 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:01:52.076800  9326 raft_consensus.cc:1217] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:52.076833  8479 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 3, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:52.118311  8480 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 3, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:52.149564 10469 raft_consensus.cc:993] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:52.150077 10469 raft_consensus.cc:1081] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:52.152755 10416 raft_consensus.cc:1064] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:52.174983 10416 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:52.176925  8578 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:01:52.182438  8578 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:52.182842  8578 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:52.188900  8578 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:52.190923  8578 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:52.193415  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:52.203850  8727 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:52.204355  8727 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:52.205246  8727 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:52.206905  8727 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:52.214741  8727 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:52.215912  8550 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:52.216770 10474 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:52.217322 10474 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:52.218159 10474 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:52.230432  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:52.231756  8655 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:52.254247  8727 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.256274 10287 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:52.267971  9326 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.269446  8480 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:52.270650 10416 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:52.274101  8655 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:52.274703  8578 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.276410 10416 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:52.301915 10416 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.314116  9329 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.346377  8480 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.411181 10423 raft_consensus.cc:1064] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:52.415064 10423 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:52.426491  8582 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:52.428347 10470 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:52.461158  8659 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:52.465631 10469 raft_consensus.cc:1064] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:52.466270  8727 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.468390 10469 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:52.470031 10468 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:52.493002  8655 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:52.503507  9326 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.506640 10468 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:52.517309  8657 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.519052 10468 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:52.526341  8582 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.528290 10351 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:52.531976  9325 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:52.533859  9329 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.534622  8728 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.539966 10474 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:52.547032 10350 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:52.556825  8731 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:52.560859 10416 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:52.576458  8652 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.596115  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:52.581903 10350 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260709 13:01:52.578809 10368 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 13:01:52.631639  8650 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:01:52.633365  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 186565us
I20260709 13:01:52.654588 10423 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:52.659024 10318 raft_consensus.cc:1064] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:52.668850 10318 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:52.672722 10468 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.684926  8657 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.689219  8731 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.691475 10501 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.702051 10475 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:52.716382  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98766.7us
I20260709 13:01:52.715040  9325 raft_consensus.cc:2955] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.722157  8652 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:52.725361 10474 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:52.743376  8657 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.755620  8728 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:52.758319  8724 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:52.765369  8658 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:52.770895 10504 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:52.801620  8480 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:52.802897  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 128650us
I20260709 13:01:52.814553  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet ea2142382a5d4439915e07c70837c688 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:52.827098  8479 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:52.840938  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 165270us
I20260709 13:01:52.853999  8575 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.860074  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ea2142382a5d4439915e07c70837c688 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:52.866541  8731 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:52.893855 10511 tablet_replica.cc:333] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:52.896992  8479 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.920081  8580 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:52.933277  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 52958.1us
I20260709 13:01:52.935200  8657 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:52.938727 10468 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:52.947441 10500 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:52.949327  8657 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:52.963527  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 90384.7us
I20260709 13:01:52.979453  8480 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:53.006522  8731 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:53.014750 10511 raft_consensus.cc:2243] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:53.015511 10511 pending_rounds.cc:70] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
W20260709 13:01:53.015348  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 113893us
I20260709 13:01:53.015942 10511 pending_rounds.cc:77] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 2 index: 4 } timestamp: 7305634251383803904 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ea2142382a5d4439915e07c70837c688" old_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } } new_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } } }
I20260709 13:01:53.017472 10511 raft_consensus.cc:2917] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted
I20260709 13:01:53.018451 10511 raft_consensus.cc:2889] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260709 13:01:53.018880 10511 raft_consensus.cc:2272] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:53.022382  8731 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.025723 10511 ts_tablet_manager.cc:1916] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:53.025604  9329 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.039785  9328 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.046314  9327 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:53.051048  9326 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:53.058907  9327 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:53.064317 10511 ts_tablet_manager.cc:1929] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:53.064846 10511 log.cc:1199] T ea2142382a5d4439915e07c70837c688 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/ea2142382a5d4439915e07c70837c688
I20260709 13:01:53.092623  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 78613f64484447c487fa71abda1ec1ac with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 13:01:53.103121  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.106765  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet ea2142382a5d4439915e07c70837c688 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:01:53.169950  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 57032.3us
W20260709 13:01:53.183758  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44612.1us
I20260709 13:01:53.189667 10475 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 13:01:53.197310 10318 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
W20260709 13:01:53.207937  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54029.7us
I20260709 13:01:53.252265  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet 78613f64484447c487fa71abda1ec1ac with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41978
I20260709 13:01:53.254565 10511 tablet_replica.cc:333] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:01:53.258311 10511 raft_consensus.cc:2243] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:53.260282 10511 raft_consensus.cc:2272] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:01:53.272202  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 77820.8us
W20260709 13:01:53.279286  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 55530.8us
I20260709 13:01:53.283368 10350 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:53.286331 10423 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:53.288672 10423 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:01:53.297901  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.309273  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.311919 10350 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.293320 10511 ts_tablet_manager.cc:1916] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:53.323014  9327 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.358448 10520 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:53.387691  9328 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.385565 10518 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.436801  8481 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.449651  8727 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:53.461997  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 5559db905acf4ba0b48e9e3e87f49c1c with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:53.468669  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5559db905acf4ba0b48e9e3e87f49c1c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:53.503446  8481 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.511837  9328 raft_consensus.cc:2955] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.526140 10511 ts_tablet_manager.cc:1929] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:53.526787 10511 log.cc:1199] T 78613f64484447c487fa71abda1ec1ac P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/78613f64484447c487fa71abda1ec1ac
I20260709 13:01:53.532298  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet 78613f64484447c487fa71abda1ec1ac (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:53.554764  8655 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:53.608557  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51714.7us
W20260709 13:01:53.612159  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 102077us
I20260709 13:01:53.625949 10504 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:53.629832 10530 tablet_replica.cc:333] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:53.632835  8728 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.653918 10530 raft_consensus.cc:2243] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:53.654995 10530 raft_consensus.cc:2272] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:53.667357 10530 ts_tablet_manager.cc:1916] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:53.671232  8724 raft_consensus.cc:2955] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.693367  8580 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:53.721557  9327 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:53.723829 10350 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:53.725183 10530 ts_tablet_manager.cc:1929] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:53.725762 10530 log.cc:1199] T 5559db905acf4ba0b48e9e3e87f49c1c P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/5559db905acf4ba0b48e9e3e87f49c1c
I20260709 13:01:53.731307  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 5559db905acf4ba0b48e9e3e87f49c1c (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:53.732577 10504 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.757021  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:53.759671  9328 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.792505  8655 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:53.794406 10518 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:53.799376  8652 raft_consensus.cc:2955] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:53.807344  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b387b386886048c8b66e5f499f02aab2 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:53.832471  8480 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index 4 to 5, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:53.851979  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet b387b386886048c8b66e5f499f02aab2 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40398
I20260709 13:01:53.893347  8729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:53.894243 10535 tablet_replica.cc:333] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:01:53.894245  8729 raft_consensus.cc:606] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:53.896962 10535 raft_consensus.cc:2243] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:53.898088 10535 raft_consensus.cc:2272] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:53.898741  8652 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:01:53.899335  8652 raft_consensus.cc:606] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:53.903657  8580 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:53.904331  8580 raft_consensus.cc:606] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:53.918875  8575 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d851286f2cf248549fc81f8296784c08"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:53.919553  8575 raft_consensus.cc:606] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:53.921206  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:01:53.913251 10535 ts_tablet_manager.cc:1916] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:53.921365 10350 raft_consensus.cc:993] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:53.922214 10350 raft_consensus.cc:1081] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:01:53.923552  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:53.929963  9328 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43640
I20260709 13:01:53.930763  9328 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:53.931154  9328 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:53.942766  9328 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:53.944962 10535 ts_tablet_manager.cc:1929] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:53.945498  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:53.945855  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:53.946056  9328 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:53.946204  8575 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:53.946425  8729 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:53.946607  8575 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:53.945650 10535 log.cc:1199] T b387b386886048c8b66e5f499f02aab2 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/b387b386886048c8b66e5f499f02aab2
I20260709 13:01:53.947469  8575 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:53.948860  8575 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:53.950228  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet b387b386886048c8b66e5f499f02aab2 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:01:53.955724  8575 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:53.956015  8729 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:01:53.957347  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:53.958726 10429 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:53.959200 10429 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:01:53.960132 10429 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:53.984503  8480 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:54.039594 10500 raft_consensus.cc:993] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:54.040187 10524 raft_consensus.cc:1081] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:01:54.041749 10287 raft_consensus.cc:993] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:54.042284 10287 raft_consensus.cc:1081] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:54.043713  8652 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:01:54.044381  8652 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:54.044729  8652 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.051908  8575 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41992
I20260709 13:01:54.052618  8575 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:54.053051  8575 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:54.057927  8652 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.060746  8575 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.063218  8575 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:54.065546  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:54.065876  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:54.066906  8659 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:54.067306  8580 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.067600  8659 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:54.068989  8659 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.073037  8659 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:54.080660  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:54.081748  8729 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:54.095221  8659 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:54.096858  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:54.097774 10518 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:54.098297 10518 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:54.099411  8580 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:01:54.099620 10518 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.100858  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:01:54.104264 10416 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:54.104843 10416 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:01:54.105880 10416 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.100147  8575 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:54.100777  8652 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:54.102546  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:54.113682  8728 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:54.114045  8728 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:54.114806  8728 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.115908  8728 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.124819  8728 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:01:54.098137  8729 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:01:54.134660  8480 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 3 to 4, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 4 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:54.137363  8479 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 3, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:54.251262 10518 raft_consensus.cc:993] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:54.251880 10350 raft_consensus.cc:1081] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:54.253705  8728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d851286f2cf248549fc81f8296784c08"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:54.254402  8728 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:54.255831  8728 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:54.263069  8728 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.269160  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:54.269737  8575 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:54.270041  8575 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:54.270856  8575 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.271797  8575 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:54.279057  8728 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:54.280223  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:54.280467  8575 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:54.280827  8652 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:54.283655  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:54.284679 10283 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:54.285122 10283 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:54.285928 10283 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:54.303757  8652 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:01:54.314585  8480 catalog_manager.cc:5697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 2, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:54.488143  8728 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:54.489926 10505 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:54.493919  8582 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:54.495276  8728 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:54.501457 10505 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:54.503690 10518 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:54.553208  8659 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:54.555285 10504 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:54.585151  8728 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:54.586515 10416 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:54.620611  8582 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:54.622252 10525 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:54.697225  8659 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:54.699620 10500 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:54.708297  8582 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:54.710062 10283 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:54.924739  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:54.956326  8582 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:54.971294  8728 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:54.971664  8659 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:54.973029 10350 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:54.974581  8548 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7c87eef075e49ba98e563e6fcc45d83. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:54.975504 10350 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:55.013983 10518 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.016384  8729 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.025727  8659 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.049723  8479 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.079455  8582 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:55.096863  8729 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:55.109238 10350 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:01:55.119819  8548 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 081470ef13974fe497f88859ada43ffa. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:55.122906  8652 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:55.188187 10350 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 13:01:55.232803 10561 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.245940  8659 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.254889  8728 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.262550  8575 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:01:55.279245  8548 consensus_peers.cc:597] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e8a9411169f407cba1b44443edc9840. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:55.293584  8724 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:01:55.267307  8481 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 3, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:55.296307 10556 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:55.318239 10518 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.339671  8724 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.349974  8659 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:55.367522  8652 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:01:55.364641  8481 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index -1 to 4, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
W20260709 13:01:55.378315  8623 consensus_peers.cc:597] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2947646ebfc34bd7a5a55312f5812ee2. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:55.378361  8730 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:55.381440 10552 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:55.391181  8580 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:55.394063 10504 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:55.397933 10416 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:55.398861  8659 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.417907 10552 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.437637  8575 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.439368  8724 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.434059 10572 ts_tablet_manager.cc:933] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:55.453783 10572 tablet_copy_client.cc:323] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:55.455931  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet a7c87eef075e49ba98e563e6fcc45d83 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:55.456529  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet a7c87eef075e49ba98e563e6fcc45d83 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-a7c87eef075e49ba98e563e6fcc45d83
I20260709 13:01:55.470484  8593 tablet_copy_source_session.cc:215] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:55.470019  8480 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 2, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:01:55.524017 10572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c87eef075e49ba98e563e6fcc45d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:55.544656 10572 tablet_copy_client.cc:806] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:55.545280 10572 tablet_copy_client.cc:670] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:55.568743 10572 tablet_copy_client.cc:538] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:01:55.606468  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60334.8us
I20260709 13:01:55.618749 10572 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:55.685855 10572 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:55.687031 10572 tablet_bootstrap.cc:492] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:55.691274 10572 ts_tablet_manager.cc:1403] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.073s	user 0.042s	sys 0.006s
I20260709 13:01:55.694742 10572 raft_consensus.cc:359] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:55.705583 10572 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:55.706497 10572 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:55.709966 10572 ts_tablet_manager.cc:1434] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 13:01:55.711942  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-a7c87eef075e49ba98e563e6fcc45d83 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:55.712463  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-a7c87eef075e49ba98e563e6fcc45d83 on tablet a7c87eef075e49ba98e563e6fcc45d83 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:55.726923 10578 ts_tablet_manager.cc:933] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:55.742388 10578 tablet_copy_client.cc:323] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:55.744267  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 081470ef13974fe497f88859ada43ffa from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:55.744858  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 081470ef13974fe497f88859ada43ffa from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-081470ef13974fe497f88859ada43ffa
I20260709 13:01:55.761703  8593 tablet_copy_source_session.cc:215] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:55.765287 10578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081470ef13974fe497f88859ada43ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:55.773126 10572 ts_tablet_manager.cc:933] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer a0d703ee18464475a290425976246555 (127.7.78.65:46779)
I20260709 13:01:55.798892 10581 ts_tablet_manager.cc:933] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:55.805541 10572 tablet_copy_client.cc:323] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.65:46779
I20260709 13:01:55.807304  8593 tablet_copy_service.cc:140] P a0d703ee18464475a290425976246555: Received BeginTabletCopySession request for tablet 9e8a9411169f407cba1b44443edc9840 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:34698)
I20260709 13:01:55.807991  8593 tablet_copy_service.cc:161] P a0d703ee18464475a290425976246555: Beginning new tablet copy session on tablet 9e8a9411169f407cba1b44443edc9840 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:34698: session id = 453e644bad5d410ab9be85b353e28dc2-9e8a9411169f407cba1b44443edc9840
I20260709 13:01:55.816898  8593 tablet_copy_source_session.cc:215] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:55.823566 10578 tablet_copy_client.cc:806] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:55.826964 10578 tablet_copy_client.cc:670] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:55.833240 10581 tablet_copy_client.cc:323] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.66:39929
I20260709 13:01:55.833539 10572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8a9411169f407cba1b44443edc9840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:55.835386  8668 tablet_copy_service.cc:140] P d0a2b9c938d94a4fa258e068ea054d84: Received BeginTabletCopySession request for tablet 2947646ebfc34bd7a5a55312f5812ee2 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:38092)
I20260709 13:01:55.835991  8668 tablet_copy_service.cc:161] P d0a2b9c938d94a4fa258e068ea054d84: Beginning new tablet copy session on tablet 2947646ebfc34bd7a5a55312f5812ee2 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:38092: session id = 453e644bad5d410ab9be85b353e28dc2-2947646ebfc34bd7a5a55312f5812ee2
I20260709 13:01:55.837180 10578 tablet_copy_client.cc:538] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:55.887017  9327 raft_consensus.cc:1217] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:55.902516  8668 tablet_copy_source_session.cc:215] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:55.910027 10581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947646ebfc34bd7a5a55312f5812ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:55.914664 10578 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:55.925700 10572 tablet_copy_client.cc:806] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:55.926364 10572 tablet_copy_client.cc:670] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:55.949643 10572 tablet_copy_client.cc:538] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:55.957005 10581 tablet_copy_client.cc:806] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:55.957876 10581 tablet_copy_client.cc:670] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:55.968418 10581 tablet_copy_client.cc:538] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:55.982264 10581 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:56.017450 10572 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:01:56.051961 10578 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:56.053020 10578 tablet_bootstrap.cc:492] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:56.059543 10578 ts_tablet_manager.cc:1403] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.145s	user 0.025s	sys 0.004s
I20260709 13:01:56.079094 10578 raft_consensus.cc:359] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:56.080090 10578 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:56.080801 10578 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:56.084122 10572 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:56.085111 10572 tablet_bootstrap.cc:492] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:56.137616 10572 ts_tablet_manager.cc:1403] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.120s	user 0.028s	sys 0.007s
I20260709 13:01:56.155898 10578 ts_tablet_manager.cc:1434] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.096s	user 0.008s	sys 0.000s
I20260709 13:01:56.161933 10581 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:56.163229 10581 tablet_bootstrap.cc:492] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:01:56.164059 10581 ts_tablet_manager.cc:1403] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.182s	user 0.029s	sys 0.005s
I20260709 13:01:56.166548  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:56.184779  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 97040.9us
I20260709 13:01:56.189878  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-081470ef13974fe497f88859ada43ffa received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:56.190399  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-081470ef13974fe497f88859ada43ffa on tablet 081470ef13974fe497f88859ada43ffa with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:56.248240 10572 raft_consensus.cc:359] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:56.249186 10572 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:56.249948 10572 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:56.236457 10581 raft_consensus.cc:359] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:56.252523 10581 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:01:56.253155 10572 ts_tablet_manager.cc:1434] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.099s	user 0.004s	sys 0.010s
I20260709 13:01:56.257941  8593 tablet_copy_service.cc:342] P a0d703ee18464475a290425976246555: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-9e8a9411169f407cba1b44443edc9840 received from {username='slave'} at 127.0.0.1:34698
I20260709 13:01:56.254691 10581 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:56.258421  8593 tablet_copy_service.cc:434] P a0d703ee18464475a290425976246555: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-9e8a9411169f407cba1b44443edc9840 on tablet 9e8a9411169f407cba1b44443edc9840 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:56.267346 10581 ts_tablet_manager.cc:1434] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.103s	user 0.007s	sys 0.001s
I20260709 13:01:56.277904  8668 tablet_copy_service.cc:342] P d0a2b9c938d94a4fa258e068ea054d84: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-2947646ebfc34bd7a5a55312f5812ee2 received from {username='slave'} at 127.0.0.1:38092
I20260709 13:01:56.278415  8668 tablet_copy_service.cc:434] P d0a2b9c938d94a4fa258e068ea054d84: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-2947646ebfc34bd7a5a55312f5812ee2 on tablet 2947646ebfc34bd7a5a55312f5812ee2 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:01:56.286144  9327 raft_consensus.cc:1217] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 13:01:56.292155  9326 raft_consensus.cc:1217] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260709 13:01:56.337662  8724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:01:56.338371  8724 raft_consensus.cc:606] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:01:56.353638  8582 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:01:56.354292  8582 raft_consensus.cc:606] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:01:56.367985  9326 raft_consensus.cc:1217] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:56.380079  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:01:56.381822  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:56.388293 10569 raft_consensus.cc:1064] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:56.391430 10569 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:56.403221  8724 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:56.404809 10590 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:56.408113  9326 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:56.408131  8652 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:56.409659 10568 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:56.411649 10568 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:56.439277 10569 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.441206  9327 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.459880  8652 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.473634  8724 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.481475  8481 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.687340  8575 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:56.702435  8724 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:56.709471 10590 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:56.720927 10569 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.733773  8724 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.740389  9327 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:56.745743 10504 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:56.750988 10283 raft_consensus.cc:993] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:56.751566 10283 raft_consensus.cc:1081] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:01:56.758415 10416 raft_consensus.cc:1064] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:56.761719  8582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:01:56.762315  8582 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:56.762691  8582 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:56.761027 10416 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:56.769922  9327 raft_consensus.cc:2955] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.779021  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a7c87eef075e49ba98e563e6fcc45d83 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:56.782514 10504 raft_consensus.cc:993] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
W20260709 13:01:56.809679  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44051.1us
I20260709 13:01:56.821661 10590 raft_consensus.cc:1081] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:01:56.824810  8724 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40408
I20260709 13:01:56.825537  8724 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:56.825961  8724 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:56.864672 10518 raft_consensus.cc:1064] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:56.871479  8582 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:56.872081  8466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 91048.1us
I20260709 13:01:56.875048  8582 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:56.879359  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:01:56.880072  8729 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:56.880364  8729 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:56.884843  8729 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:56.886122  8729 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:56.883294  8482 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 4 to 5, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:01:56.905910  8724 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:56.913450  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:01:56.925325  9326 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:56.940351  8729 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:56.946138  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:01:56.965389  8724 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:56.965659  8652 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:56.970275  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:56.970785  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet a7c87eef075e49ba98e563e6fcc45d83 with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38080
I20260709 13:01:56.980126  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:01:56.980396 10590 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:56.980871  8578 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:56.980973 10590 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:56.981236  8578 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:01:56.982175  8578 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:56.983448  8578 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:57.014065  9328 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 LEARNER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:57.029978 10416 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:57.039508  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 132036us
I20260709 13:01:57.040103  8724 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:57.047734  8652 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:01:57.047860  9326 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:01:57.011551 10590 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:57.052229  8578 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:01:57.054980  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:01:57.061184 10606 tablet_replica.cc:333] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:01:57.062825 10606 raft_consensus.cc:2243] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:57.075608 10589 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:57.056212 10518 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:57.077720 10589 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:01:57.078996 10589 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:57.054065 10552 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 13:01:57.089876  8582 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:57.129976 10606 raft_consensus.cc:2272] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:57.221104 10600 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20260709 13:01:57.387708 10610 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 13:01:57.389312  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:57.407544  8479 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:57.441574 10613 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:01:57.441978 10518 raft_consensus.cc:1064] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:01:57.446123 10606 ts_tablet_manager.cc:1916] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:57.444664 10518 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:57.509068  8724 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:01:57.546125  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 310867us
W20260709 13:01:57.560892  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 339889us
W20260709 13:01:57.684746  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 150058us
I20260709 13:01:57.731443  8575 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:57.764027  8480 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 4, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 4 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:57.872876  8652 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:01:57.903198  8622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.040s	sys 0.063s
W20260709 13:01:57.908000  8622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.041s	sys 0.063s
W20260709 13:01:57.908584  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 689632us
I20260709 13:01:57.926050 10625 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:57.926908 10625 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:57.929863 10625 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:01:57.936215  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 188347us
W20260709 13:01:58.004572  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49268.3us
I20260709 13:01:58.008527 10416 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:58.013439  9328 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:58.009532 10416 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:01:58.019057 10416 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:58.054399  9329 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:58.066695  9329 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEARNER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:58.076339  8733 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:58.083370  8729 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:58.086344  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:57.983615 10518 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 13:01:58.093770  8658 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:58.124795  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:58.145040  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110774us
I20260709 13:01:58.111931 10621 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 13:01:58.161172  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:58.166461 10632 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 13:01:58.136467  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:58.223835 10590 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:58.206496  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 52722.8us
I20260709 13:01:58.160997 10555 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:01:58.165700 10637 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:58.229722 10637 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:58.161840 10568 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:01:58.213759  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 52828.6us
I20260709 13:01:58.130592 10620 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:01:58.248553 10634 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
W20260709 13:01:58.251312  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 239827us
I20260709 13:01:58.252990  8582 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:58.263798  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 109710us
I20260709 13:01:58.267933  8652 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:58.270612 10606 ts_tablet_manager.cc:1929] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:58.271088 10606 log.cc:1199] T a7c87eef075e49ba98e563e6fcc45d83 P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/a7c87eef075e49ba98e563e6fcc45d83
I20260709 13:01:58.271842 10429 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:58.272228 10606 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38080 (request call id 81) took 1307 ms. Trace:
I20260709 13:01:58.272657 10429 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:58.276325  8625 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:58.284438 10429 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:58.285933  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:58.240046 10622 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 13:01:58.254361 10590 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 13:01:58.291455 10637 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:58.309262 10621 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:58.251520 10620 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:58.272755 10606 rpcz_store.cc:276] 0709 13:01:56.964298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.964936 (+   638us) service_pool.cc:224] Handling call
0709 13:01:57.051091 (+ 86155us) ts_tablet_manager.cc:1151] Deleting tablet a7c87eef075e49ba98e563e6fcc45d83
0709 13:01:57.053279 (+  2188us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 13:01:58.078743 (+1025464us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:58.270480 (+191737us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:58.272124 (+  1644us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31164,"tablet-delete.queue_time_us":71260,"thread_start_us":45174,"threads_started":1}
I20260709 13:01:58.282038 10600 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:58.321563 10620 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:58.320634 10642 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:58.467891  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 199977us
W20260709 13:01:58.468874  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 215157us
I20260709 13:01:58.476212  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:58.476325  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 193258us
I20260709 13:01:58.506647 10504 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:58.507433 10504 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:58.513696  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:58.516007  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:58.519551 10605 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:58.520110 10605 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:58.535940 10504 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:58.536989 10605 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:01:58.584995  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 111430us
W20260709 13:01:58.586843  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 296550us
I20260709 13:01:58.610249 10628 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:01:58.615347  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 347886us
I20260709 13:01:58.653143  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:58.615213  9328 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:01:58.659201  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 72975.8us
I20260709 13:01:58.661552 10283 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:58.662467  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:58.659183  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:58.664214 10633 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:58.611073 10628 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:58.680382  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:58.686741  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 411014us
I20260709 13:01:58.711870  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:58.713413 10628 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:58.759815  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:01:58.822248  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 123333us
I20260709 13:01:58.827448  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:58.830019 10608 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:01:58.831292 10608 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:58.867527  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 123988us
I20260709 13:01:58.873327  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:01:58.912011  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 238364us
I20260709 13:01:58.924552 10608 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:58.940738  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:58.941900  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:58.997807  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:59.013688  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:01:59.016232  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:01:59.026149 10601 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:59.033980 10643 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:01:59.034651 10643 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:59.036231 10675 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:59.057750 10588 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:59.073912 10555 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:59.100167 10646 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:59.073993 10601 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:01:59.080310 10638 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.906s	user 0.007s	sys 0.016s
W20260709 13:01:59.119042 10639 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.035s	user 0.016s	sys 0.048s
W20260709 13:01:59.119563 10639 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.051s	user 0.016s	sys 0.048s
I20260709 13:01:59.079535 10555 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:01:59.134903  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 341557us
W20260709 13:01:59.147068  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 76246.6us
I20260709 13:01:59.156126 10643 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:59.158161  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:59.155004  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:59.174005 10626 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:59.185256 10676 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:59.185936 10676 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:59.174935 10626 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:59.117897 10646 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:59.128268 10601 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:59.233415  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1551) took 1561 ms. Trace:
I20260709 13:01:59.233891  8732 rpcz_store.cc:276] 0709 13:01:57.605995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.668145 (+ 62150us) service_pool.cc:224] Handling call
0709 13:01:57.668662 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.669017 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.669038 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:57.670364 (+  1326us) log.cc:844] Serialized 548 byte log entry
0709 13:01:57.944155 (+273791us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:57.944198 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.944266 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:59.079869 (+1135603us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b240038f9a8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:01:59.154535 (+ 74666us) raft_consensus.cc:1685] finished
0709 13:01:59.154558 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:59.167027 (+ 12469us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24853376,"thread_start_us":273326,"threads_started":1,"wal-append.queue_time_us":454960}
I20260709 13:01:59.248374  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:59.238538 10647 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:01:59.240752 10608 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:01:59.260880 10682 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 13:01:59.263046 10668 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:59.268093 10601 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:59.236315 10670 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:59.275892  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:59.284322  8577 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:01:59.270324  8481 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 302) took 1258 ms. Trace:
I20260709 13:01:58.594367  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet a7c87eef075e49ba98e563e6fcc45d83 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:01:59.286165  8468 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 691453us
I20260709 13:01:59.252594 10555 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:59.287065  8481 rpcz_store.cc:276] 0709 13:01:58.011831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.011995 (+   164us) service_pool.cc:224] Handling call
0709 13:01:58.086976 (+ 74981us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b540012a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b80090 00007f1409c1ef3a 00007f140f9d51cc 00007f140fad337e 00007f140df942a4 00007f140df94232 00007f140df94161 00007f140df940dc 00007f140df93392 00007f1407f5148c 00007f1407f508c6
0709 13:01:59.270255 (+1183279us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":11341184}
I20260709 13:01:59.296155 10668 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:59.321499 10646 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:59.322280 10630 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:59.322902 10668 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:59.326743 10665 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:59.327486 10665 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:59.280356 10626 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:59.280721 10608 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:59.322964 10630 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:59.378253  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:01:59.384246 10631 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:59.297217 10647 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:59.391961  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:59.385267 10631 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:59.336690 10676 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:59.412545 10601 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:59.413254 10601 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:01:59.415498 10601 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:59.425503 10665 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:59.442627 10525 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:59.443303 10525 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:01:59.401896 10647 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:01:59.546481  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:01:59.548482  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:59.463430 10607 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:01:59.552542  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 84291.3us
I20260709 13:01:59.459695 10608 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:01:59.578115 10617 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:59.580628  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:01:59.542309 10630 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:01:59.578780 10617 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:01:59.778487  8575 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.040s	sys 0.062s
W20260709 13:01:59.950762  8575 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.041s	sys 0.070s
W20260709 13:01:59.951979  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 371301us
W20260709 13:01:59.859625  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.028s	sys 0.037s
W20260709 13:01:59.953071  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.032s	sys 0.039s
I20260709 13:01:59.868867  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:59.954157  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 705307us
I20260709 13:01:59.584820 10632 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:59.906621 10661 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
W20260709 13:01:59.961779  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 367428us
I20260709 13:01:59.640959 10625 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:01:59.638998 10675 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:59.992455  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:59.953313  8732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.009s	sys 0.018s
I20260709 13:01:59.677714 10640 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:59.733655 10669 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:59.853773 10603 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:59.718207 10602 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:01:59.985668  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:59.985805  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:01:59.986452 10604 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:59.987807 10646 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:01:59.964545  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:01:59.993435  8732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.009s	sys 0.018s
W20260709 13:01:59.820366 10683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.013s	sys 0.043s
I20260709 13:01:59.839649 10630 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:59.641796 10608 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:01:59.698549 10657 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:00.007091  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.007434 10640 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.009093  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:59.993657  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.009547  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.010751  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:01:59.596176 10607 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.013867  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.016194  8731 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:00.032282  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.025604 10589 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.010176 10669 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:00.018676 10667 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.002s	sys 0.024s
W20260709 13:02:00.084916  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 521884us
W20260709 13:02:00.034559 10683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.013s	sys 0.046s
W20260709 13:02:00.116925  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 144679us
I20260709 13:02:00.036885 10668 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.034242 10552 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:00.090149 10667 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.006s	sys 0.024s
I20260709 13:02:00.126348  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:00.029769 10625 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.034533 10604 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.075309 10630 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.102713 10700 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:00.137887  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 103344us
I20260709 13:02:00.039810 10602 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.103502 10589 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.037263 10603 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.154953 10589 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:00.156047 10602 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.158178 10602 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:00.158684 10602 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.159197 10602 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:00.149765  8549 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.031s	sys 0.084s
I20260709 13:02:00.037876 10646 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.156138 10603 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.085729 10608 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:00.161386  8549 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.031s	sys 0.084s
I20260709 13:02:00.124961  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:00.139242 10552 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:00.181777  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56954us
W20260709 13:02:00.182727  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 229670us
W20260709 13:02:00.186137  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 626356us
W20260709 13:02:00.191845  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 163056us
I20260709 13:02:00.143100 10668 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.186375 10589 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:00.240077 10429 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.105806 10657 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.204622 10590 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.240839 10429 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.225762 10622 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:00.255921  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 103324us
I20260709 13:02:00.233439 10707 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.233033 10605 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:00.289283  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 162939us
W20260709 13:02:00.291568  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 98654.9us
I20260709 13:02:00.168521 10700 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.248040 10640 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.248617 10617 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.346802 10429 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.344384  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:00.355094  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 149385us
I20260709 13:02:00.255940 10590 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:00.402227  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 159149us
I20260709 13:02:00.386523 10626 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:00.342352 10605 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.390889 10669 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.416043 10603 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:00.439770  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.441951  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:00.291468 10622 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:00.435668 10696 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.303s	user 0.000s	sys 0.001s
W20260709 13:02:00.452989  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 74467.9us
W20260709 13:02:00.419832  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 197744us
I20260709 13:02:00.456444 10612 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:00.457448  8698 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:00.463977  8575 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
W20260709 13:02:00.469844  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 167851us
I20260709 13:02:00.472659 10604 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.475893 10625 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.478458 10590 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.479970 10657 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.355686 10707 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.458787 10626 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:00.465445 10607 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.498888 10283 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:00.500627  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 1956) took 1125 ms. Trace:
I20260709 13:02:00.501488  8732 rpcz_store.cc:276] 0709 13:01:59.375570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.376954 (+  1384us) service_pool.cc:224] Handling call
0709 13:01:59.395443 (+ 18489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:59.404423 (+  8980us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:59.404467 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:59.409973 (+  5506us) log.cc:844] Serialized 548 byte log entry
0709 13:02:00.031212 (+621239us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:00.031252 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.031314 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:00.500336 (+469022us) raft_consensus.cc:1685] finished
0709 13:02:00.500366 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.500540 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":620843,"threads_started":1,"wal-append.queue_time_us":668270}
I20260709 13:02:00.501355 10283 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:00.514041 10701 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.260s	user 0.000s	sys 0.002s
I20260709 13:02:00.515359 10662 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:00.517112  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.517633 10700 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.524026  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.492794 10702 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:00.463486 10603 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.530452 10707 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.537137 10624 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:00.543097  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 85919.9us
I20260709 13:02:00.523420 10612 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:00.551918  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51634us
W20260709 13:02:00.551920  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 86081.7us
I20260709 13:02:00.560607 10667 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.561213 10667 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.561796  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.562453  9326 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:00.563619  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.567118  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:00.566812  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.522419 10626 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.551297 10630 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.592099  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.590871  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.593758  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.594149  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.594817  8579 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:00.580425  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.593938 10709 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.591639  8652 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:00.589521  9331 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:00.570859 10604 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.459833 10659 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:00.538354 10624 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.623025  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:00.588766 10712 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.183s	user 0.000s	sys 0.002s
I20260709 13:02:00.549052 10702 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.625138  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.616631  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.633611  8656 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:02:00.637704  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 64214us
I20260709 13:02:00.583459 10607 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.639266  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.590392 10707 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:00.599658  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.609658 10608 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.612039 10603 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.650532  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.610838 10605 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.607167 10706 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:00.661594  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.662513  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:00.617004 10604 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.665902 10605 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:00.651489 10707 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.666477 10605 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.666257  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.668346 10707 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.634660 10709 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.680773 10626 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.681129  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.675007 10603 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:00.686292  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.686292 10631 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.689638  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.690129  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.664829 10622 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.691516  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.668910 10605 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.693470  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.648680 10607 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.696363  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.698426 10646 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.693245 10656 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.665185 10706 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.662828 10602 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.687980 10617 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.583544 10612 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.726271 10636 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:00.726229 10720 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.704986 10706 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:00.716219 10724 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.706571  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.671695 10688 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:00.702612 10702 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.700063 10656 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.727188 10636 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.717355 10617 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.739283 10504 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.716321  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.844579 10504 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.810274  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.847807  8579 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:00.748142 10724 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.882627  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.884325 10724 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.884938 10724 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.886065  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.888029  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.897533 10604 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.916316  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2079) took 1256 ms. Trace:
I20260709 13:02:00.917096  8653 rpcz_store.cc:276] 0709 13:01:59.659352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.914894 (+1255542us) service_pool.cc:224] Handling call
0709 13:02:00.915387 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.915640 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.915684 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:00.915753 (+    69us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:00.915822 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.915930 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.916091 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:00.925617  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.926591  9328 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:00.929942  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.930611  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:00.931255  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.932227  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.932929  8733 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:00.933857  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.934530  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.937028  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:00.846086 10700 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.939312  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.837860 10621 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:00.851894 10504 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.950691 10709 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.959064  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.833509 10624 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:00.890146 10622 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:00.967072 10623 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:00.980638 10622 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.982915 10622 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.748865 10720 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.983726  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:00.988314 10658 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:00.951851 10630 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:00.995666  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.997012  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.998337  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.925976 10724 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.003487  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.006165  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.007129  8731 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:01.008667  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.968755 10525 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.015969 10636 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.978585  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:00.992820 10617 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:01.021668  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.957852 10700 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:00.994122 10602 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:00.993639  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:00.999212 10657 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:00.989044 10658 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.003489 10630 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.070369 10700 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:01.071552 10658 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:01.074580 10630 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.075634 10630 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:01.076385 10630 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:00.965677 10621 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.086344 10652 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:01.054075 10607 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.102633 10656 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:01.103714 10666 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:01.084474 10631 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:01.109766 10631 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.055903 10657 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.095932 10624 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:01.113643 10720 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.116158  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:01.130800  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 245109us
I20260709 13:02:01.133013  8728 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:01.105026  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:01.155421  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.105587 10652 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.125958 10612 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.123072 10624 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.149549 10604 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:01.175726  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.106559 10666 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.160629  8550 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:01.188681  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:01.189713  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.176738 10724 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:01.205173  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 268128us
I20260709 13:02:01.181602 10630 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.192678 10670 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:01.197954 10621 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.219527  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:01.224570  9331 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:01.202489 10614 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:01.177771 10604 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.236449  8480 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 298) took 3668 ms. Trace:
I20260709 13:02:01.227885  9297 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:01.229538  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 218493us
I20260709 13:02:01.230540 10614 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:01.227739 10602 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:01.281193  9297 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:01.282087  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:01.283946  8657 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:01.285424 10659 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.602s	user 0.005s	sys 0.008s
I20260709 13:02:01.255832 10614 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:01.289863  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46760.3us
I20260709 13:02:01.290849  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:01.266249 10623 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:01.292547  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:01.296295  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:01.297782  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:01.295794 10730 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:01.281857 10589 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:01.304947  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:01.305639  8656 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:01.306715  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:01.294035 10702 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:01.302958 10607 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:01.309674 10720 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:01.310333 10720 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.310253 10589 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.312614 10720 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:01.310889 10728 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 13:02:01.282280 10602 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.297278 10603 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:01.324076 10603 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:01.324752 10603 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:01.324752  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 1985) took 1674 ms. Trace:
I20260709 13:02:01.325431  8650 rpcz_store.cc:276] 0709 13:01:59.650296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.651981 (+  1685us) service_pool.cc:224] Handling call
0709 13:01:59.699868 (+ 47887us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.081000 (+381132us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.081055 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:00.081149 (+    94us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:00.081208 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.324243 (+1243035us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540026c354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:01.324301 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.324547 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":875482880}
I20260709 13:02:01.316910 10612 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:01.327144 10612 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.317607 10714 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:01.330727 10714 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:01.331127 10714 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:01.331434 10602 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.331005  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.321161  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.332276  8724 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:01.236922  8480 rpcz_store.cc:276] 0709 13:01:57.568455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.568615 (+   160us) service_pool.cc:224] Handling call
0709 13:02:01.236374 (+3667759us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:57.831670 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:57.832533 (+   863us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:57.832578 (+    45us) write_op.cc:435] Acquired schema lock
0709 13:01:57.832602 (+    24us) tablet.cc:662] Decoding operations
0709 13:01:57.833109 (+   507us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:57.833180 (+    71us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:57.833223 (+    43us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:57.907667 (+ 74444us) tablet.cc:701] Row locks acquired
0709 13:01:57.907694 (+    27us) write_op.cc:260] PREPARE: finished
0709 13:01:57.942789 (+ 35095us) write_op.cc:270] Start()
0709 13:01:57.942920 (+   131us) write_op.cc:276] Timestamp: P: 1783602117913659 usec, L: 0
0709 13:01:57.942943 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:01:58.006370 (+ 63427us) log.cc:844] Serialized 535 byte log entry
0709 13:01:59.120795 (+1114425us) op_driver.cc:464] REPLICATION: finished
0709 13:01:59.733754 (+612959us) write_op.cc:301] APPLY: starting
0709 13:01:59.733844 (+    90us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:59.733953 (+   109us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:59.798555 (+ 64602us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:59.854278 (+ 55723us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:59.874477 (+ 20199us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:59.907913 (+ 33436us) write_op.cc:312] APPLY: finished
0709 13:01:59.941171 (+ 33258us) log.cc:844] Serialized 23 byte log entry
0709 13:01:59.953048 (+ 11877us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:59.957237 (+  4189us) write_op.cc:454] Released schema lock
0709 13:01:59.964729 (+  7492us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:59.991415 (+ 26686us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:59.991455 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":612689,"apply.run_cpu_time_us":3005,"apply.run_wall_time_us":257646,"num_ops":1,"prepare.queue_time_us":61092,"prepare.run_cpu_time_us":6201,"prepare.run_wall_time_us":193436,"replication_time_us":1177716,"thread_start_us":299140,"threads_started":1}]]}
I20260709 13:02:01.334255  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.341095  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.355568  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.386507  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:01.388366  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 49226.6us
I20260709 13:02:01.341964  8731 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:01.301286 10729 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:01.394708  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.299990 10713 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:02:01.341904 10688 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.005s
I20260709 13:02:01.318100 10604 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.396767  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.320152 10667 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:01.405964  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.410379  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.411676  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.329273 10612 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:01.316632 10652 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:01.398327  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 112339us
I20260709 13:02:01.473690  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.554216  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:01.678570  8575 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:01.610338  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.614054  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.655660  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.701952  8726 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:01.535727  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.705513  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.731112  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.704768  8576 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
W20260709 13:02:01.596719 10737 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.183s	user 0.000s	sys 0.000s
W20260709 13:02:01.512698 10706 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.774s	user 0.013s	sys 0.000s
I20260709 13:02:01.319370 10630 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.214s
I20260709 13:02:01.741622  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2081) took 1761 ms. Trace:
I20260709 13:02:01.319224 10623 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:01.746860  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.404913 10713 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:01.755259  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.419902  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:01.757829  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:01.605814 10733 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:01.435165  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:01.535778 10702 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:01.577780 10650 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:01.760179  8733 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:01.762615 10429 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:01.806914 10702 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.807283 10429 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.596967 10614 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.798565 10733 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.720647 10666 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.765604  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:01.731061 10612 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:01.790490  9324 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:01.823151 10612 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.824890 10614 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.783272 10736 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:01.776091 10603 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.807031 10650 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.755625  8654 rpcz_store.cc:276] 0709 13:01:59.980043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.980211 (+   168us) service_pool.cc:224] Handling call
0709 13:02:00.025120 (+ 44909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.154427 (+129307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.154454 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:00.154491 (+    37us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:00.154524 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.733251 (+1578727us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540018b0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:01.733297 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.741509 (+  8212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296820992}
I20260709 13:02:01.810835  8724 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:01.770015  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:01.862901  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92935us
I20260709 13:02:01.864344  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:01.848505 10742 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:01.775125 10713 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:01.875519 10742 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:01.759297 10706 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.852180 10738 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:01.768373 10623 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:01.845052  8733 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:01.877918 10741 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:01.939239  8579 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:01.887665 10646 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:01.958639 10706 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.861384 10736 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:01.987412  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 123099us
I20260709 13:02:01.988518  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:01.993352 10429 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:01.993443  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:01.964705 10612 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:01.997864  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:02.000478  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:02.001618 10429 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.002235 10429 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:01.986457 10714 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.655s	user 0.013s	sys 0.000s
I20260709 13:02:01.934353 10738 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.931313 10742 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:01.996894 10603 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:01.997557 10602 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:02.001438 10642 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:01.997671 10604 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:02:02.403127  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 396997us
I20260709 13:02:02.415221 10429 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:02.428263  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.006673 10733 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:02.030817 10739 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 13:02:02.442426 10429 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.010865 10609 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.015882 10741 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:02.424237 10742 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:02.489555 10624 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.470832 10429 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.491719 10429 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:02.391286 10706 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:02.492674 10624 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.317937 10702 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.027048 10623 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:02.425729 10604 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:02.485584 10609 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.486701 10748 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.487946 10750 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.388669 10614 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:02.472461 10733 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:02.475919  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49071.4us
W20260709 13:02:02.494719  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.015s	sys 0.014s
I20260709 13:02:02.006340 10714 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.475621 10738 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:02.285655 10745 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.018102 10736 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.493377 10706 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.029670 10744 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.480054  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:02.283576 10621 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:02.498530  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 497894us
W20260709 13:02:02.508325  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.015s	sys 0.014s
W20260709 13:02:02.513760  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 829453us
I20260709 13:02:02.498075 10604 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:02.500092 10748 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.505728 10614 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.517820  8729 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:02.524879  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:02.523980 10663 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:02.493434 10624 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.524165 10602 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.506553 10750 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.540052  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:02.524365 10642 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:02.512962 10733 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.518667 10621 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:02.514752 10744 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.513255 10745 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.547257 10755 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.547734 10748 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.549775 10614 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.541528 10602 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.528841 10753 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.529537 10752 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.575361 10752 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:02.575980 10602 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.577808 10624 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.550853 10642 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:02.551236 10609 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.532575 10663 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.554497 10668 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.554675 10714 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.555352  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:02.536614 10736 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:02.594179 10642 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.594750 10736 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.595741 10736 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:02.558858 10706 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:02.567215  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47810.2us
I20260709 13:02:02.540598 10702 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.574018 10745 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.574596 10621 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.580556  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:02.581105 10753 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.585883 10733 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.593580 10609 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.596379  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:02.597378  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58562.2us
I20260709 13:02:02.598399  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:02.676095  8578 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:02.600461 10713 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.692s	user 0.012s	sys 0.000s
W20260709 13:02:02.676829  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 78519.1us
I20260709 13:02:02.599036 10702 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.675015 10609 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.679129  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:02.680476  8696 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:02.681558 10733 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:02.677733  8697 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.553s (SENT) [suppressed 3 similar messages]
I20260709 13:02:02.570379 10755 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.687724  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:02.679642 10752 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:02.686868 10756 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:02.690532 10762 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:02.694025 10714 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.678779 10742 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:02.696846  8730 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:02.706950 10750 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:02.707224  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:02.724150  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:02.725361  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:02.728029 10604 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:02.733031  8732 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:02.734822  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:02.734989 10663 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:02.710762  8625 connection.cc:256] RPC call timeout handler was delayed by -1.1597518789999981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2215s for a response
W20260709 13:02:02.742925  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65195.3us
I20260709 13:02:02.722960 10525 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:02.686064 10713 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:02.749214  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 572) took 2498 ms. Trace:
I20260709 13:02:02.749745  9329 rpcz_store.cc:276] 0709 13:02:00.252823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.252940 (+   117us) service_pool.cc:224] Handling call
0709 13:02:01.321952 (+1069012us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:01.322177 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.964424 (+642247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.964478 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.964548 (+    70us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:01.964600 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.748960 (+784360us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54003b01d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:02.749005 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:02.749139 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1505869952}
I20260709 13:02:02.756508  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 607) took 1536 ms. Trace:
I20260709 13:02:02.722252 10714 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:02.930670  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:02.807997  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:02.962250  8657 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:02.814318  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:02.955538  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.969926 10283 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.970333  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:02.739809 10668 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.976418  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:02.978449 10283 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:02.807834 10755 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.975901 10745 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.983837 10745 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.982332 10283 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.757751 10702 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:02.986784 10745 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.904798 10736 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:02.745828 10663 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.759198 10604 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.987396  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.020763  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.047119  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.047344  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.048468  8652 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:02.765686 10602 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:03.049991  8732 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:02.987315 10283 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:02.987526 10614 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:03.051585 10716 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:02.756559 10753 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.054703 10646 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.860468 10740 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:02.904345 10761 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:02.811390 10750 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:02.792652 10631 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.059587  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.065608  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.071981  8582 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:02.792435 10713 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.067741  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.070763  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:03.077304  9329 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.081324  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:02.762189 10525 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.067896 10614 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.089982 10761 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.077301  8654 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:02.757216  9323 rpcz_store.cc:276] 0709 13:02:01.219731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.220827 (+  1096us) service_pool.cc:224] Handling call
0709 13:02:02.754881 (+1534054us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:02.755171 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:02.755937 (+   766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:02.755988 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:02.756054 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:02.756110 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.756204 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:02.756384 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9288576}
I20260709 13:02:03.108624  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.109015  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.001866 10668 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.113669  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.092278  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.114687  9331 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
W20260709 13:02:03.097119  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 366657us
I20260709 13:02:03.067262 10552 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:02.991963 10621 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:03.116309  8654 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:03.116094  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.117074  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.125419  8582 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:03.119318  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:02.996384 10671 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:03.118175  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:03.126236  8577 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:03.128224  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.128994  8732 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:03.068962 10646 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.067684 10602 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.069003 10663 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.075277 10716 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.126870  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:03.139137  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.140504  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:03.131824  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.135385 10761 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.132216  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.129082 10755 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:03.127439 10621 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.134819  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.071751 10740 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.078745 10750 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:03.141247  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 1973) took 2197 ms (client timeout 1696 ms). Trace:
I20260709 13:02:03.142935  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:03.143010  8732 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:03.141952 10552 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:03.155757  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.152850  8625 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:03.150233  8727 rpcz_store.cc:269] 0709 13:02:00.932639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.933422 (+   783us) service_pool.cc:224] Handling call
0709 13:02:03.116908 (+2183486us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:03.130104 (+ 13196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6458112}
I20260709 13:02:03.158982 10740 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.130831 10671 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.135406 10614 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.162848 10716 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.135501 10735 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:03.164915 10750 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.166704 10621 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.167003  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:03.167245 10605 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:03.168056 10525 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.161360  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.170382  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2099) took 1918 ms. Trace:
I20260709 13:02:03.170702 10602 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.172482 10668 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.174052 10646 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.159621 10589 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.184341  8731 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.186233  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.150648 10603 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.155911 10604 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:03.159014 10650 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.198402  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.242317 10603 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:03.242785  8652 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:03.242988 10603 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.211123  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.193684  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.195592  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.246470  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.246513  8577 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:03.247237  8582 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:03.195859  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.189441 10740 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:03.193637  8658 rpcz_store.cc:276] 0709 13:02:01.251588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.251791 (+   203us) service_pool.cc:224] Handling call
0709 13:02:01.252092 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.254609 (+  2517us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.265704 (+ 11095us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.265755 (+    51us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:01.265791 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.170114 (+1904323us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:03.170175 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.170317 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4931456}
W20260709 13:02:03.193567  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 145844us
W20260709 13:02:03.213426  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46446.9us
I20260709 13:02:03.248441  8651 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:03.251277  9297 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:03.261148  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:03.271354  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 118527us
I20260709 13:02:03.272199  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.266805  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:03.192209 10714 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.276692  8625 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:03.277576  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.256438 10766 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:03.280049  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.290917  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.269569 10750 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:03.269879 10589 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:03.296662  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.284003 10636 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.306797  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.307580  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.300669  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.308104  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.308301  8584 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:03.308756  8576 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:03.309777  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.198275 10605 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.277944 10658 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:03.316277  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.317101  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.211658 10735 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.285523 10766 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:03.300364 10767 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:03.310391  8580 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:03.299396  8625 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:03.306828  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.308777  8482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 197) took 3086 ms. Trace:
I20260709 13:02:03.273669 10757 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:03.314582 10589 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.329198 10757 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.330999 10589 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.331408 10757 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.299451  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:03.303555  8729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1569) took 3643 ms (client timeout 2215 ms). Trace:
I20260709 13:02:03.327211 10766 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:03.304637  8479 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 302) took 5932 ms. Trace:
I20260709 13:02:03.298365 10750 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:03.343806  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.317706  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.317744  8576 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:03.320781  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.346127  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.304988 10636 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:03.314086  9325 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
W20260709 13:02:03.332141  8729 rpcz_store.cc:269] 0709 13:01:59.660103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.660264 (+   161us) service_pool.cc:224] Handling call
0709 13:02:01.320992 (+1660728us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:03.303321 (+1982329us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540042fed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:03.303472 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2764193920}
I20260709 13:02:03.295735  9323 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:03.333298  9329 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:03.300546 10688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.631s	user 0.016s	sys 0.033s
W20260709 13:02:03.366133 10688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.737s	user 0.016s	sys 0.034s
I20260709 13:02:03.351431  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.367403  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.319278  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.357746  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.396220  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.397416  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:03.322288 10658 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.398465  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.399324 10658 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.359701  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.359934 10663 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.362255  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:03.406389  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44167.7us
I20260709 13:02:03.407440 10605 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.347289 10750 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:03.408174 10604 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.408664 10604 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:03.409039 10604 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.362596 10671 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.363631  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.364884  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:03.353814 10636 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.460196  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.373303  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 1972) took 2441 ms. Trace:
I20260709 13:02:03.385336  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.327181 10767 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:03.358484  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.403117  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.463726 10767 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.403570 10657 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.407528  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.456925  8652 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:02:03.473701  8652 raft_consensus.cc:606] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:03.465005  8658 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.461054  8730 rpcz_store.cc:276] 0709 13:02:00.932284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.932410 (+   126us) service_pool.cc:224] Handling call
0709 13:02:02.688265 (+1755855us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:02.688424 (+   159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:02.696754 (+  8330us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:02.731024 (+ 34270us) log.cc:844] Serialized 14 byte log entry
0709 13:02:02.855244 (+124220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:02.855319 (+    75us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:02.855381 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.372266 (+516885us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b18003c8500. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419
0709 13:02:03.372975 (+   709us) spinlock_profiling.cc:243] Waited 8.09 ms on lock 0x7b540008acd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:03.373048 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.373192 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":138,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":11647,"spinlock_wait_cycles":43191552,"thread_start_us":123696,"threads_started":1,"wal-append.queue_time_us":242289}
I20260709 13:02:03.475919  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.477069  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.478662  8730 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:03.480005  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.481670  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.465219  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.466838  8656 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.467728  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.428516  8698 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:03.486974  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58576.4us
I20260709 13:02:03.418012  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.487907  8733 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:03.488670  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 196314us
I20260709 13:02:03.489019  8729 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:03.461050 10736 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.470563  8655 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.385291 10744 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.387331 10735 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.378341 10609 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.491561  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.348791 10772 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:03.485491  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:03.498646  9330 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
W20260709 13:02:03.500305  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 1985) took 1754 ms (client timeout 1718 ms). Trace:
W20260709 13:02:03.500806  8724 rpcz_store.cc:269] 0709 13:02:01.742626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.755010 (+ 12384us) service_pool.cc:224] Handling call
0709 13:02:03.496326 (+1741316us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b54004727d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:03.496444 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75443456}
I20260709 13:02:03.502439  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.502660 10774 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:03.503995 10774 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:03.504464 10774 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:03.505194  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:03.491768 10657 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:03.506971  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:03.514813  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:03.516024  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.516798  9328 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:02:03.506690 10736 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.514418 10657 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:03.517789  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.518329 10617 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:03.521260  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.522015 10617 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:03.522682 10617 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.513868 10772 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:03.523483 10657 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.528020 10772 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.514951 10769 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.533172 10766 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.533776 10604 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.519457 10736 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:03.536312 10769 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:03.535748 10766 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.538044 10769 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.537762 10736 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.505342 10774 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.543931 10750 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.546344 10750 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.547528 10657 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.540729 10774 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.547971 10736 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.551182 10736 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.552055 10767 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.554934 10767 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.556571 10769 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.310941  8481 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 2 to 3, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:03.328068  8482 rpcz_store.cc:276] 0709 13:02:00.222595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.224011 (+  1416us) service_pool.cc:224] Handling call
0709 13:02:03.308718 (+3084707us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3048473,"reported_tablets":1}
I20260709 13:02:03.333606  8479 rpcz_store.cc:276] 0709 13:01:57.372041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.372189 (+   148us) service_pool.cc:224] Handling call
0709 13:02:03.304578 (+5932389us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:57.422529 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:57.514681 (+ 92152us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:57.514730 (+    49us) write_op.cc:435] Acquired schema lock
0709 13:01:57.514753 (+    23us) tablet.cc:662] Decoding operations
0709 13:01:57.750744 (+235991us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:57.762569 (+ 11825us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:57.785685 (+ 23116us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:57.813930 (+ 28245us) tablet.cc:701] Row locks acquired
0709 13:01:57.813962 (+    32us) write_op.cc:260] PREPARE: finished
0709 13:01:57.829831 (+ 15869us) write_op.cc:270] Start()
0709 13:01:57.830011 (+   180us) write_op.cc:276] Timestamp: P: 1783602117829760 usec, L: 0
0709 13:01:57.830046 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 13:01:57.831040 (+   994us) log.cc:844] Serialized 530 byte log entry
0709 13:01:58.068112 (+237072us) op_driver.cc:464] REPLICATION: finished
0709 13:01:58.958421 (+890309us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f1406b0c0ac 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f140d64cef0
0709 13:01:59.198702 (+240281us) write_op.cc:301] APPLY: starting
0709 13:01:59.200932 (+  2230us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:59.210183 (+  9251us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:59.222146 (+ 11963us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:59.236408 (+ 14262us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:59.250681 (+ 14273us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:59.253953 (+  3272us) write_op.cc:312] APPLY: finished
0709 13:01:59.271190 (+ 17237us) log.cc:844] Serialized 23 byte log entry
0709 13:02:00.132394 (+861204us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:00.138568 (+  6174us) write_op.cc:454] Released schema lock
0709 13:02:00.143074 (+  4506us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:00.178522 (+ 35448us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:00.180167 (+  1645us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1090383,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1108432,"apply.run_cpu_time_us":66274,"apply.run_wall_time_us":965956,"num_ops":1,"prepare.queue_time_us":278,"prepare.run_cpu_time_us":15568,"prepare.run_wall_time_us":409582,"replication_time_us":221476,"spinlock_wait_cycles":30687616,"thread_start_us":1880645,"threads_started":2,"wal-append.queue_time_us":834553,"wal-append.run_cpu_time_us":2614,"wal-append.run_wall_time_us":1119568}]]}
W20260709 13:02:03.609632  8550 connection.cc:256] RPC call timeout handler was delayed by -1.3119687549999526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
I20260709 13:02:03.609961  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.350186  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.390444  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.613008  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:03.613180  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.615697  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.618356  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.506026  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.513631  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:03.625391  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.652468  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.645287  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:03.645274  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:03.667128  8581 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.636302  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.670554  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.633154  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.671478  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:03.653721  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.650609  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.667496  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.659188  9328 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.674285  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.646826  8578 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.629809  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1511) took 2643 ms. Trace:
W20260709 13:02:03.641916  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.675100  8579 rpcz_store.cc:276] 0709 13:02:00.986082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.986234 (+   152us) service_pool.cc:224] Handling call
0709 13:02:01.010399 (+ 24165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.946997 (+936598us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.629071 (+1682074us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b5400343ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:03.629587 (+   516us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2134679808}
I20260709 13:02:03.637209  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.647128  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.678001  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.652976  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.678963  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.669270  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.679385  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.645444  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:03.679495  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.680239  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.655586  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.657919  9329 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.660964  9330 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.651631  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.678951  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:03.667966  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.653594  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.683215  8733 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:03.648732  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.625255  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.665633  8657 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:03.740491  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:03.689437  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.662009  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.628227  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:03.729550  8578 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:03.690253  8580 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:03.765875 10671 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:03.765672 10429 consensus_queue.cc:1175] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1dc41db601fe4b91aa6d461b504da0e7" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 3 index: 6 } }
I20260709 13:02:03.766249 10769 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.766752 10671 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.755438  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.785022 10698 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:03.785712 10698 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:03.785743  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 144606us
I20260709 13:02:03.786304 10698 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.788121  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:03.789474 10429 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:03.789907 10697 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:03.790503 10697 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:03.790983 10697 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:03.792115  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 113424us
I20260709 13:02:03.796002  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.796845  8584 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.801764  8652 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:03.802680  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.793287 10604 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.809376 10614 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.809885 10614 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:03.810451 10614 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.811148  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.812139  9324 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:03.816489  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:03.817008  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.817732 10657 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.815413  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.819303  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.818442 10620 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:03.821038 10620 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:03.821874  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.829697  8581 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:03.807868  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.834396  8576 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.829139  8658 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:03.833680  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.831532  9329 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.846415  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:03.847522  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:03.849666 10671 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.850451  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.850840  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:03.852321  8728 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:03.822479 10620 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.857724  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:03.857376  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.863865  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.865934 10769 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:03.864807  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.871326  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.875880 10617 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.878966 10617 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:03.889387  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.890661  8728 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:03.890307 10614 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.892812  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.893229  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.898021  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.900642  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:03.901512  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.904891  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.906961  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.908218  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:03.917339  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 85133us
I20260709 13:02:03.919562  8658 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.912369  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:03.923635  8653 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:03.927601  8657 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:03.918401  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.933876  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.934705  8655 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:03.935065  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:03.937754  8654 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.938413  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.939666 10525 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:03.940025  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:03.941191 10646 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:03.942282  8576 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.942555 10646 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:03.943941 10646 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:03.946877 10603 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.687s	user 0.011s	sys 0.015s
I20260709 13:02:03.947876 10646 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.949710 10636 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.947551 10603 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.955018  8651 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:03.955646  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:03.956945 10604 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:03.958040  8659 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:03.956951  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:03.961143 10614 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.961971 10603 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:03.965552 10636 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:03.957597 10604 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:03.973899  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:03.974468 10604 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:03.975997 10688 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.976529 10688 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:03.977070 10688 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.977587  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:03.978222 10666 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:03.978888 10697 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:03.980036 10657 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:03.980468 10657 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:03.981814  8583 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:03.983067  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:03.957530  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 49892.7us
I20260709 13:02:03.984030  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:03.982556  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53132.9us
I20260709 13:02:03.999284  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:04.002131  8576 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:03.982582 10621 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:03.972981 10698 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.006723  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:03.981372 10697 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:04.006871 10614 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.017603  8578 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.022044 10429 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:04.022540 10429 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:04.022941 10429 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:04.024994  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:04.026091  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:03.981593 10657 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.035466 10283 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.036020 10283 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.036422 10283 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.037636  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.038749  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.014387 10666 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:02:04.050191 10666 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.050616 10666 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:04.047904 10657 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.053071  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:04.051764 10758 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 13:02:04.055220 10769 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.055881 10758 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.056407 10758 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:04.059686 10671 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.061224 10671 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.061744 10671 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:04.064370 10774 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:04.044571 10603 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.065311 10603 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.065843 10603 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.066680  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.067790 10652 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.068202  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.069088 10652 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.059693 10429 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.049095 10688 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.081861 10774 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.083155 10429 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:04.083838  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45131.3us
I20260709 13:02:04.086658 10603 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.089972 10614 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:04.093068 10614 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.093199 10758 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.095110 10758 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.096302 10614 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.091776  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.097282  8578 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:04.098260  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:04.062861 10671 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.081095 10688 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.051724 10666 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.083173 10774 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.111979  8578 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:04.112742 10525 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:04.113248 10525 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.113719 10525 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:04.114472 10603 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:04.119378  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 82277.9us
I20260709 13:02:04.112025 10666 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.119804 10671 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.099915  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.124428 10620 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:04.128587 10769 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:04.124428  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.130015 10769 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:04.130124 10620 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.130411 10769 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.131275 10620 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.131379 10769 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.133020 10769 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:04.128101  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:04.134011 10617 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:04.134572 10617 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.135672  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.135556 10617 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.136023 10766 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.137657  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.137984 10646 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:04.139768 10429 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.065393 10698 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.141510 10283 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.144692  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.147707  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.156386 10429 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:04.158510  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54275.5us
I20260709 13:02:04.159363  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.159544  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:04.160219 10652 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.160487 10603 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:04.155465 10688 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:04.160991 10603 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:04.161510 10603 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:04.162043  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:04.162359  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:04.162824  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 130674us
I20260709 13:02:04.170179 10429 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:04.172232  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 72433.3us
W20260709 13:02:04.173357  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.171298 10646 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.176554 10525 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.177661  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.178609  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.178366 10603 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.184448 10766 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.184623 10652 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.185783  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:04.176259  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:04.187737 10652 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.189975  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.190616  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.194927 10688 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.195461 10688 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:04.195953 10688 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.188804 10603 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:04.194051 10604 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.201974 10604 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.202036 10429 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.202457 10604 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.202884 10429 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.203421 10429 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.206272 10525 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.209343 10283 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.211493  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.212258 10603 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.214308  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.210796 10657 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.221459  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:04.221567 10657 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:04.223047  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.223198  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.224349 10652 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.224742 10652 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:04.223723  9329 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:04.225741 10652 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.231789  9329 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:04.232523  9326 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:04.233618 10698 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.234861  9326 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:04.235081  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:04.238147  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48280.9us
I20260709 13:02:04.239017  9329 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.241178 10429 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.247776 10688 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.239037 10283 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.243526  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.241899  9326 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.246982  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.257153 10283 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:04.267956 10787 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:04.268168 10283 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.271169  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.272143 10642 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.275743  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.278327  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.278630 10429 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:04.255744 10688 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:04.269771 10787 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:04.281495  8583 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:60002
I20260709 13:02:04.291491  8583 raft_consensus.cc:606] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.280087  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.292886 10787 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:04.298499  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.290220 10758 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:04.308449  9326 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.312657  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.303001 10758 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:04.340865 10688 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.342424 10614 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.342860 10614 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:04.343786 10614 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:04.344935  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 98026.8us
I20260709 13:02:04.344422 10283 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.311446 10666 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.352044  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.328979 10758 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:04.314735 10525 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.358877 10283 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.366519  8583 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.368031  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:04.374720  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62138.1us
I20260709 13:02:04.375131 10429 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.375763  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.381862  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.371733 10614 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:04.397809  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 182882us
I20260709 13:02:04.401846 10429 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:04.402482 10429 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.403378 10657 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.404067 10657 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.406526  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.407248  8583 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:04.393045 10620 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.409698 10620 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.411072  8481 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 303) took 5061 ms. Trace:
I20260709 13:02:04.410677 10620 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.411589 10283 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.413501 10666 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:04.414268  8479 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 3 to 4, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:04.424628  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.424947  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:04.420128  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44458us
I20260709 13:02:04.426235  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.427983  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.425155  8479 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 4 to 5, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:04.429332  8576 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:04.418218 10525 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.435554  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.432004 10666 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:04.436851 10666 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:04.438231 10790 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:04.440030 10790 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:04.441104 10790 consensus_queue.cc:1243] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:04.442183 10657 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.442728 10657 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:04.443286 10657 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.443388 10652 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.447852  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:04.416431  8481 rpcz_store.cc:276] 0709 13:01:59.349357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.354542 (+  5185us) service_pool.cc:224] Handling call
0709 13:01:59.550244 (+195702us) spinlock_profiling.cc:243] Waited 64 ms on lock 0x7b540012a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b80090 00007f1409c1ef3a 00007f140f9d51cc 00007f140fad337e 00007f140df942a4 00007f140df94232 00007f140df94161 00007f140df940dc 00007f140df93392 00007f1407f5148c 00007f1407f508c6
0709 13:02:04.410892 (+4860648us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:03.610739 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:03.785054 (+174315us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:03.785119 (+    65us) write_op.cc:435] Acquired schema lock
0709 13:02:03.785138 (+    19us) tablet.cc:662] Decoding operations
0709 13:02:03.788398 (+  3260us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:03.788514 (+   116us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:03.788552 (+    38us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:02:03.788869 (+   317us) tablet.cc:701] Row locks acquired
0709 13:02:03.788919 (+    50us) write_op.cc:260] PREPARE: finished
0709 13:02:03.789102 (+   183us) write_op.cc:270] Start()
0709 13:02:03.789279 (+   177us) write_op.cc:276] Timestamp: P: 1783602123789033 usec, L: 0
0709 13:02:03.789310 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 13:02:03.830381 (+ 41071us) log.cc:844] Serialized 591 byte log entry
0709 13:02:04.090204 (+259823us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b18001662a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:04.206046 (+115842us) op_driver.cc:464] REPLICATION: finished
0709 13:02:04.395948 (+189902us) write_op.cc:301] APPLY: starting
0709 13:02:04.401477 (+  5529us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:04.405614 (+  4137us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:04.405660 (+    46us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:04.406006 (+   346us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:04.406343 (+   337us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:02:04.406389 (+    46us) write_op.cc:312] APPLY: finished
0709 13:02:04.406664 (+   275us) log.cc:844] Serialized 23 byte log entry
0709 13:02:04.406885 (+   221us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:04.406963 (+    78us) write_op.cc:454] Released schema lock
0709 13:02:04.407018 (+    55us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:04.407427 (+   409us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:04.407466 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1544147,"reported_tablets":2,"spinlock_wait_cycles":147166080,"child_traces":[["op",{"apply.queue_time_us":154564,"apply.run_cpu_time_us":2336,"apply.run_wall_time_us":33175,"num_ops":1,"prepare.queue_time_us":18722,"prepare.run_cpu_time_us":27256,"prepare.run_wall_time_us":320616,"raft.queue_time_us":112940,"raft.run_cpu_time_us":8046,"raft.run_wall_time_us":74863,"replication_time_us":416549,"spinlock_wait_cycles":97456512,"thread_start_us":218745,"threads_started":4,"wal-append.queue_time_us":86451}]]}
I20260709 13:02:04.451079 10429 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:04.451305 10657 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.452206 10603 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:04.452653 10603 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:04.442174 10792 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 13:02:04.453749 10603 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.454816 10652 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.455171 10603 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:04.456893  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.457315 10525 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:04.457753  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.456930 10620 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.458479  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.459326 10604 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.459849  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.454882 10792 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:04.461026 10792 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:04.462184  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.462081 10792 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.463832  8582 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.464461  9322 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:04.457953 10525 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:04.468920  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.469345 10525 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:04.448997  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:04.471006  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 121432us
I20260709 13:02:04.457257  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.471829  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.473453 10283 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.474341  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.474931  8582 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.478463  8583 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:04.479204  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.448616 10758 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.482683 10429 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.488411  8576 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:04.494176  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 73459.8us
I20260709 13:02:04.494495 10795 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.502051 10646 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:04.505594 10646 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.506189 10794 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:04.506852 10794 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.507274 10794 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.507759 10646 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.511598  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.513121  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.513152  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.513854  8582 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:04.514245 10657 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:04.514700  8578 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:02:04.526057  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 77107.4us
I20260709 13:02:04.527674 10652 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.503201 10604 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:04.531730  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:04.534307  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.535503 10789 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.536350 10789 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:04.536849 10789 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:04.537833 10789 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.545011  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.544847 10604 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.546428 10604 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.547314 10604 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.547753  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.547930 10604 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:04.548954  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.549013 10795 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.470399 10525 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.549134 10604 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.547358 10787 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.556288  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:04.563762  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:04.542284 10666 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.567941 10758 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.568356 10646 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.568995 10666 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:04.570919  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.571398 10652 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.574576  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.578290 10604 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.578912  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:04.580389  8725 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.580983  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.581734  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.604311  8583 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:04.583683  8727 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.583983 10792 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:04.605775 10429 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.585209  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.586747  8724 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:04.567984 10525 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:04.587352 10787 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:04.617825  8732 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:04.618582  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47724us
I20260709 13:02:04.620147  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.624029 10603 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.592953  8726 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:04.603845 10794 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.584250  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:04.625175  8481 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:04.582728 10283 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.626665 10283 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:04.627211 10794 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:04.627444 10283 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.628823 10283 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.628938 10429 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:04.630056 10429 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.630365 10429 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.631335 10429 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.632491 10769 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:04.633899 10429 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.627262  8481 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 3, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:04.636588  8481 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 3 to 4, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:04.604897 10795 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:04.610682  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.638290  8481 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:04.639671 10642 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:04.640110 10642 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:04.640924 10642 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.642153 10642 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:04.643355  8583 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:04.611184  8654 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.645604  8583 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:04.611444 10774 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:04.611546  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.646849 10774 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.611706  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.647220 10774 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.647526  8659 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:04.612143 10688 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:04.612304  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.648267  8732 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:04.609679  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 227856us
I20260709 13:02:04.648175 10774 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.650673 10774 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.651985  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.654318  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.635221  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.658705 10688 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:04.659827 10688 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.655567  9324 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:04.662125  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:04.663295 10769 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:04.663558 10766 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:04.663817  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:04.665524  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 135352us
I20260709 13:02:04.665984  8655 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:04.669040  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:04.672279  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 177123us
I20260709 13:02:04.672531  8726 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:04.673065  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:04.672996  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.673255 10283 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:04.678140  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:04.680050  8729 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:04.684543  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.688835 10283 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.673107 10604 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:04.696002  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.698694  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.699744  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.701254  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:04.679109 10792 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.698745  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.709203  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.714457  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:04.715703 10283 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.717123  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.718007  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:04.718210  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.719211  8729 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.720914  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:04.721312  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 86183.6us
I20260709 13:02:04.722172  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.684765 10795 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 13:02:04.723402 10795 consensus_queue.cc:1243] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:04.723944 10795 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.724119 10790 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:04.724514 10790 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:04.724978  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.725466 10790 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.726924  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.729023  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.729689 10798 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:04.729882  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.730948  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.731187  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.728297  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.723973  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:04.743554  8650 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:04.745117  8727 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.751211  8726 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:04.752182  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:04.698966 10646 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 13:02:04.751655  8731 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.758323  8732 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:04.761593  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56333.1us
I20260709 13:02:04.765606  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.726966 10790 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:04.777635  8727 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.758309 10642 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:04.785228  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61293us
I20260709 13:02:04.787256  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.798632  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.803449  8727 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:04.809811  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.773624 10793 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:04.820493  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44496us
I20260709 13:02:04.823104 10429 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.822911  8582 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.825284 10766 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 13:02:04.829598 10766 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.830124 10766 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:04.828263 10796 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.833695  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.833832  8725 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 13:02:04.835203  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46618.3us
I20260709 13:02:04.836019  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.829507  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.837597 10429 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:04.824818  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.837364 10793 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.841342  8579 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.846925 10603 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:04.847443 10603 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:04.847674 10793 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:04.848557  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.848351 10603 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.849639  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:04.851756  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.852241  8654 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:04.850035 10603 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.853636  8658 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:04.854285  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.854207  8651 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:04.874195  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49438.5us
I20260709 13:02:04.876093  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.882923  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.861045  8481 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:04.867503 10798 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.889860  8481 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 3, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:04.857360 10642 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:04.894873  8656 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:04.899324  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.903523  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.906036  9323 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:04.909154  9329 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:04.857648  8653 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:04.914516  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.914942  9326 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:04.913532  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.917098  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.917465 10796 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:04.910198  8479 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 3, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:04.904892 10789 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.907087 10791 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 13:02:04.922379  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.880288 10793 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:04.951607  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53672.4us
W20260709 13:02:04.954123  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:04.955456  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48418.4us
I20260709 13:02:04.913622 10790 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.959059  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.960731  8654 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:04.959748  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.959755  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.962186  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:04.964636  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.855741 10766 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.950626 10642 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:02:04.970266 10789 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:04.970920 10789 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:04.971973 10789 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:04.975744  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:04.976848  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.969185 10646 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:04.980216 10766 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:04.980342  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.985983  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:04.986208  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:04.987999 10801 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 13:02:04.990777 10798 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:04.991372 10798 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:04.997277  8655 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:05.003031 10283 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:05.005975  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:05.008420  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:05.009768  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:05.010421  8658 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.012274  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:04.963474 10793 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.015214 10793 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:04.993711 10798 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:05.015604  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:05.016343  9324 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:05.020924  8654 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:05.021240 10525 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:05.023501 10525 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:05.024472 10525 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.025811 10525 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.029016  8657 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:05.031697  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:05.034761  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:05.039184  8659 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:05.063755  8731 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:05.064208  8653 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:05.065091  8724 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:05.067970  8652 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:05.033298  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:05.071259  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:05.066097  8698 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.074363  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.080614  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:05.081840 10283 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:05.082438 10283 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:05.062371  8658 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:05.088088  8657 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:05.086936  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:05.089771 10642 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:05.070227 10798 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:05.096287  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:05.104559  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:05.078900 10652 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:05.118462 10283 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:05.128988  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 215376us
I20260709 13:02:05.131055  8659 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.088282 10614 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 13:02:05.115602 10604 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:05.079295 10791 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 13:02:05.138478 10614 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.114073 10603 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:02:05.142302  8654 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:05.124185 10795 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:05.088037 10796 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.103797 10811 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.103610 10798 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:05.171722  8728 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 13:02:05.171980  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 137230us
I20260709 13:02:05.179994  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.156086 10604 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:05.188585  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 134991us
I20260709 13:02:05.189836  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.167079 10798 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:05.190598  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:05.194337  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.198132  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:05.199930  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 238754us
I20260709 13:02:05.208057  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:05.200042  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:05.183314 10791 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 13:02:05.199507  8550 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:05.223874  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.197713 10811 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.198644 10792 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.196640 10617 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:05.197031 10819 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:05.220764 10652 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.218393 10688 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.233961 10796 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.238149 10774 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:05.284581  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 85122.2us
W20260709 13:02:05.276679  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.195050 10818 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
W20260709 13:02:05.293639  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 95544.4us
I20260709 13:02:05.275560 10797 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:05.256824 10819 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:05.302655  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:05.305594  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.296782 10698 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:05.257036 10792 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.273181 10790 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 13:02:05.311932 10819 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:05.349089  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 149058us
I20260709 13:02:05.352311  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.297240 10652 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:05.329480 10823 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.098s	user 0.001s	sys 0.001s
I20260709 13:02:05.326545 10809 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:05.316569 10698 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:05.319352 10820 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 13:02:05.298929 10604 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:05.424876  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 126803us
I20260709 13:02:05.426112  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:05.381242 10795 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:05.415416 10698 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.433858 10812 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:05.458926  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.468298  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.433902 10617 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:05.475069 10811 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.431070 10795 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.476024 10834 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:02:05.485589  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 133336us
I20260709 13:02:05.486462  8728 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 13:02:05.479177  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 169987us
I20260709 13:02:05.498461  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:05.486734 10809 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.506791 10795 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:05.510946 10652 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:05.511673 10792 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:05.488463 10617 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:05.494998 10774 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:05.512905 10819 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.497933 10831 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:05.500859 10834 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:05.513757 10835 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:05.521963  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:05.529742  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:05.542322  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.542728 10646 raft_consensus.cc:993] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: : Instructing follower 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:02:05.543326 10646 raft_consensus.cc:1081] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Signalling peer 1dc41db601fe4b91aa6d461b504da0e7 to start an election
I20260709 13:02:05.543224 10816 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:02:05.545096 10698 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.547549 10698 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:05.549647 10834 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.554814 10603 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:05.555382 10835 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:05.555557 10603 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:05.555871 10835 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:05.530521 10820 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 13:02:05.557090 10835 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.564082 10835 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.562286  8728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:40420
W20260709 13:02:05.564579  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.580718  8728 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:02:05.569491  9324 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:02:05.570539 10811 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:05.570232 10617 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.534546  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:05.576184 10825 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:05.565681 10839 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:05.581302  8728 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.622395 10819 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:05.564616 10698 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:02:05.592061  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71231.2us
I20260709 13:02:05.608503  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:05.609864  8654 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:05.615799 10798 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.557019 10603 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:05.632287  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 105028us
I20260709 13:02:05.633140  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.633303  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:05.590245  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65664.1us
I20260709 13:02:05.623494 10816 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.587821 10811 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:05.643409 10811 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.629449 10839 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:05.648473 10839 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.649085  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:02:05.647603 10789 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 13:02:05.539094 10831 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.628314  8728 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.588867 10792 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260709 13:02:05.664815 10796 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:02:05.636041  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.636471 10283 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:05.629230 10698 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:05.640870 10816 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.651986  8731 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:05.663578  8726 raft_consensus.cc:606] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Received request to transfer leadership to TS 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:05.634619  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:05.691545 10832 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.214s	user 0.004s	sys 0.001s
I20260709 13:02:05.693925 10698 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.695068 10816 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:05.696429  8724 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:05.696978 10831 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:05.699245  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64692.5us
I20260709 13:02:05.629055 10825 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:05.700024  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:05.700963  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 67892.9us
I20260709 13:02:05.701886  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.707343  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:05.695087 10698 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.712922 10815 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:05.713696 10815 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:05.715063  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:02:05.714633 10815 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.716845  8731 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:05.723291  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:05.687723 10834 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.725123  8655 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:05.730214 10283 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:05.736455  8732 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:05.700258 10774 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:05.773547 10766 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:05.736001 10809 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:05.793573 10812 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.802624  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:05.807986  8732 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:05.804813 10825 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.806131  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:05.806257 10798 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:05.755178 10614 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:05.792579 10774 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.810357  8724 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
W20260709 13:02:05.814930  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 113055us
W20260709 13:02:05.816380  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 91709.7us
W20260709 13:02:05.821658  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 121664us
I20260709 13:02:05.787262 10816 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.828843  8730 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:05.829190 10820 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 13:02:05.865556  8728 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:05.850292  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:05.868786  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46346.6us
I20260709 13:02:05.884303 10652 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:05.885004 10652 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.894745 10839 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:05.896633 10652 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:05.835955 10614 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:05.901000  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50795.7us
I20260709 13:02:05.818372 10642 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260709 13:02:05.920477  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.894799 10698 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.939752  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.911064 10603 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:05.838197 10816 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.937291 10815 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:05.942194 10825 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:05.959312  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.837818 10812 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.943651 10688 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:05.948221 10774 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:05.948451 10769 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
W20260709 13:02:05.967751  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 140696us
I20260709 13:02:05.956902 10698 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.974938  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.982692  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:05.985144 10818 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.986779 10835 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:05.989465 10856 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.990059 10856 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:05.992303 10856 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:05.994626 10794 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:05.995169 10794 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:05.996312 10794 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.997998  8481 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 3, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260709 13:02:06.025139  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.039543  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.971016 10766 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:05.963389 10614 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:05.966166 10815 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.079849  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:06.081429 10853 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.081864 10851 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.969080 10820 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:06.108868  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 105093us
I20260709 13:02:06.097254 10698 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.099653 10794 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:06.111348  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:06.124043  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:06.136288 10774 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.136909 10774 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.112843 10831 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:02:06.159228  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.138729 10849 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.241s	user 0.002s	sys 0.001s
W20260709 13:02:06.172246  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48278.6us
I20260709 13:02:06.141484 10816 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.112615 10851 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.116138 10698 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.128334 10842 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:06.158725 10791 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.175554 10820 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:06.137614 10809 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 13:02:06.114243 10853 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:06.226675  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.215298  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:06.231585 10642 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.232194 10642 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.235252 10698 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.236160 10642 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.219147 10819 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:06.207863 10812 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.238184 10819 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:06.239691 10853 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.240511 10819 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:06.219462 10858 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 13:02:06.206867 10791 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.246781 10820 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:06.247712 10820 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.250280 10790 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:06.250934 10790 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:06.252164 10790 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:06.254253 10834 raft_consensus.cc:2764] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:02:06.220204 10818 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 13:02:06.256554 10818 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:06.256938 10818 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:06.257751 10818 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.259105 10818 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:06.259961 10851 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.261747 10815 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:06.262768 10847 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.346s	user 0.003s	sys 0.000s
I20260709 13:02:06.179011 10766 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.265901  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.265892 10795 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.267781  9329 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:06.268019 10795 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:06.270355 10795 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.271273 10774 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.274176  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:06.280002 10859 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 13:02:06.281558  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.282037  8657 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:06.282510  8653 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 2: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:06.289892 10820 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.293823  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.299669 10283 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:06.300364 10283 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:06.302675  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.303246 10283 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.306037  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.306743  8730 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:06.303781 10283 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.307837  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.308010 10283 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.308502  8729 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:06.309759 10790 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:06.310257  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.314277  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:06.317924  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.318890  8724 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:06.319681  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.336580  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.338595  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.340535  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:06.360898 10283 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.361824 10283 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:06.365855  8729 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:06.372057  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.372987  8730 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:06.381177  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:06.383157  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.382882  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.385198  8728 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:06.258319 10791 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.395653  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.396571  8579 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:06.383013  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.400677  8582 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:06.471313  9329 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:06.474426  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.475307  8732 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:06.382774 10646 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:06.477689  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 160079us
I20260709 13:02:06.478688  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.479439  8732 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:06.482345  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 94581.2us
I20260709 13:02:06.467697  8479 catalog_manager.cc:5697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 3, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:06.503912  8732 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:06.390146 10819 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:06.476421  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 261135us
I20260709 13:02:06.509006  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.509810  8730 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:06.461563  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.511452  8625 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:06.511519  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.514490  8732 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:06.474213 10604 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.513748  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.420822 10857 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.516970  8728 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:06.514629  8579 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:06.518782  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:06.520280  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.477515 10815 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.506412  9324 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:06.434195 10791 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.532557  9329 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:06.480630  9322 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:06.518240 10857 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.544930  8582 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 13:02:06.533088  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.521328  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.551414  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.552351  9330 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:06.552712  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.466097 10828 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:06.554705  8659 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 2: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:06.555542  8728 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 13:02:06.505125 10646 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.562288  8657 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:06.563386 10791 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.564689 10646 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.564810 10614 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.566236 10646 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:06.566262 10614 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.567092 10646 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.568665 10614 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.570330  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.571075  9322 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:06.572067  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.531653 10792 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.572700  9328 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:06.574694  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62500.2us
I20260709 13:02:06.545157 10839 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:06.521118 10604 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:06.579392  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67997.3us
I20260709 13:02:06.516921 10819 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.548026 10857 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.614930  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.615833  8728 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:06.560729 10642 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.631183  8732 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:06.531001 10815 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:06.637223  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.638878  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:06.638085  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:06.641274  8582 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:06.638046  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:06.638295  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.649019  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:06.654330  8577 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.631455  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:06.651443  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:06.651800  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:06.639241  9331 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:06.652210  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:06.656288  8576 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:06.638123  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:06.658465 10820 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:06.658857 10820 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.659175 10820 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:06.662133 10671 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:06.578006 10646 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:06.665722  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.665984  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:06.668686  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 89234us
I20260709 13:02:06.582888 10839 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:06.671293  8579 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:06.672442  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.667593  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:06.675117  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.674054  8651 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:06.681866  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.629629 10642 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.681866  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.692698  8728 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:06.681083  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:06.584062 10792 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.648218 10870 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:06.644057 10614 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:06.676245  8652 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:06.688385  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:06.669440 10603 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.704671  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:06.710413  9328 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:06.676626 10812 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.690982 10769 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.675685  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:06.741987  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 105541us
I20260709 13:02:06.724099  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.729619  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.722497  8578 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:06.697794 10642 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.734562  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.711609 10816 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.749017  8729 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:06.749146  8582 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:06.742908  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:06.747519  8651 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:06.733657 10769 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.706884 10831 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:06.730928 10812 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.763291  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.753794 10854 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.714725 10614 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:06.754379 10766 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:06.754668 10862 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
W20260709 13:02:06.755522 10869 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:02:06.755626 10604 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.752485 10816 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.755911 10875 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.756687  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:06.723232 10603 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.723680 10617 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:06.751437  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.753690  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:06.727195 10794 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:06.808914  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.810281  8584 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:06.858232 10809 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.930563 10842 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:06.932566 10836 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:06.932890 10873 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.933197 10836 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.930888 10809 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:06.937511  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 183850us
I20260709 13:02:06.791512  9328 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:06.938939  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:06.872982  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.751307 10769 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.781800  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:06.784934 10812 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.746130 10839 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.785487 10831 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.751724 10863 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.412s
I20260709 13:02:06.946583 10794 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.949791  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:06.950754  8654 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:06.746706 10867 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:06.963433  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:06.967427  8654 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:02:06.933488 10868 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.157s	user 0.001s	sys 0.001s
I20260709 13:02:06.991042  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:06.786736 10854 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.994568  8651 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:06.811030 10796 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.882866  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:06.998049  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:07.006474  8579 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:06.950214 10603 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.938411 10842 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:07.009063 10842 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.010835 10842 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:06.933385 10879 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:07.013774 10879 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:06.935649 10816 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.949314 10620 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:07.021842 10620 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.026753 10620 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:07.017390 10879 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:07.023612 10604 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.032612 10879 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:07.032688 10604 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:07.036130 10604 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:07.023998  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 256113us
I20260709 13:02:06.935858 10836 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.937175 10614 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:06.765676 10688 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.236s
I20260709 13:02:06.762945 10811 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.232s
I20260709 13:02:06.751605 10853 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.239s
I20260709 13:02:06.846910 10766 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.996361 10854 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:07.038031 10789 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.012s	sys 0.038s
I20260709 13:02:07.044679 10877 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:07.065645  8578 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:07.066663  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:07.057464  9326 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.006s	sys 0.015s
W20260709 13:02:07.067484  9326 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.006s	sys 0.015s
W20260709 13:02:07.059140  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 378153us
I20260709 13:02:07.046741 10791 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.041992 10819 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:07.069501  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 341843us
I20260709 13:02:07.046123 10815 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.072333 10876 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:07.072641 10878 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.072717  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.073419 10853 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:07.077060  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.054581 10831 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:06.961899 10875 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.955315 10873 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:07.091251  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.085420  8584 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:07.062261 10789 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.013s	sys 0.038s
I20260709 13:02:07.080122 10871 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.098748  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:07.087006  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 148091us
I20260709 13:02:07.077904 10792 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.037495 10796 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:06.990969 10867 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:07.095832  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:07.102373  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.264681  8576 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:07.135087  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:07.231858  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 291315us
I20260709 13:02:07.265573  8724 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:07.265985  8652 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
W20260709 13:02:07.266985  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.267562  8729 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:07.142752  8579 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:07.315363  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:07.316737 10789 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:07.319029 10873 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.320561 10886 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:07.305869  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.306265  8652 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:07.323220  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:07.275414  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.087723 10853 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:07.270354  8577 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.000s
I20260709 13:02:07.329833  8577 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:07.274370  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.332664  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:07.333601  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.336383  8576 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:07.132807 10790 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.823s	user 0.008s	sys 0.004s
I20260709 13:02:07.350329  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:07.351292  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.351636  8653 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:07.352114  8730 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:07.352607  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:07.091614 10809 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.336525  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:07.354308 10790 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:07.340101  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:07.356045  8650 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:07.337387  8577 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:07.261883 10642 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.000s
I20260709 13:02:07.258508 10816 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:07.229655 10798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.010s	sys 0.000s
I20260709 13:02:07.363382  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.095698 10876 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:07.360399  8650 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:07.216007 10791 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:07.223448 10814 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:07.371209 10791 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:07.138377 10872 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.366045  8584 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:02:07.361613  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.002s	sys 0.013s
W20260709 13:02:07.373819  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.002s	sys 0.013s
I20260709 13:02:07.367630  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:07.262897 10820 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.603s	user 0.012s	sys 0.000s
I20260709 13:02:07.376214  8724 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:07.378819  8729 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:07.354264  8653 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:07.373083  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:07.383543  8584 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:07.383991  8655 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:07.312471 10769 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:07.458635  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50890.2us
I20260709 13:02:07.295034 10879 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
W20260709 13:02:07.397485  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64781.3us
I20260709 13:02:07.397601  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.387507 10873 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:07.454394 10792 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:07.455014 10875 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:07.382055 10798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.015s	sys 0.000s
I20260709 13:02:07.465466  8580 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:07.362103 10889 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:07.467785  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:07.469509  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.517772  8581 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:07.518880  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.519656  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.519474  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:07.520300  8581 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:07.371397 10814 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.335520 10871 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.482353  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.091737 10878 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:07.512075  9331 raft_consensus.cc:2368] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4.
I20260709 13:02:07.470717  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:07.455545 10884 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:07.461293 10604 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.517685  9322 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:07.522424  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.470893  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:07.477738 10865 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.801s	user 0.002s	sys 0.002s
I20260709 13:02:07.468380 10800 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:02:07.534026  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:07.542973  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 72316.3us
I20260709 13:02:07.381521 10642 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:07.485474 10812 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.542s	user 0.005s	sys 0.006s
I20260709 13:02:07.511579 10816 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.384793 10877 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.111s
I20260709 13:02:07.531441  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:07.544378  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:07.387844 10820 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:07.546519  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 155479us
I20260709 13:02:07.532953 10881 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:07.551088  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.389272 10872 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.541229 10886 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:07.542609  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:07.538022 10889 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:07.539146 10796 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.541091 10604 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.562330  8659 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:07.563675 10893 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:07.470168 10892 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260709 13:02:07.564534 10890 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:07.574698 10891 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.564668 10878 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.575428 10891 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.566855  8582 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:07.561959  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.567016  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.584067  8577 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:07.550931 10798 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:07.557018 10875 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:07.553159 10617 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.547670 10876 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.565755 10854 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.577082  8652 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:07.456019 10853 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.562940 10886 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:07.593889  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.593802  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.595875  8580 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:07.554247 10812 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.565321 10881 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:07.579533 10603 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260709 13:02:07.572247 10789 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.603955  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:07.587467 10828 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.031s	user 0.009s	sys 0.004s
I20260709 13:02:07.594411 10886 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:07.626376 10819 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:07.626955 10819 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.548815 10884 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.587131 10875 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.582100 10891 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.605849  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:07.593811 10617 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.610275  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:07.610714 10867 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.611272 10642 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.612680 10769 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.613029 10604 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.548589 10671 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:07.614645  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.615324 10814 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.616446  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 626) took 1071 ms. Trace:
I20260709 13:02:07.617074 10871 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.603809 10603 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:07.619043 10872 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:07.623636 10857 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.008s	user 0.007s	sys 0.005s
I20260709 13:02:07.661648 10857 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.576647 10836 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:02:07.662771  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:07.640625 10617 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:07.641076 10812 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.663405  9322 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:07.641371  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 481) took 1007 ms. Trace:
I20260709 13:02:07.638895 10766 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.642448 10854 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.639430 10791 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.676848 10873 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:07.637195 10889 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:06.948922 10698 consensus_queue.cc:1175] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "453e644bad5d410ab9be85b353e28dc2" responder_term: 2 status { last_received { term: 2 index: 4 } last_committed_idx: 4 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)" } } last_received_current_leader { term: 2 index: 4 } }
I20260709 13:02:07.629089 10789 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:07.642510  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:07.644142  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.644238 10884 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.645310 10820 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.645978  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:07.646183 10794 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.646219  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:07.654065  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.636499 10875 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:07.658491  9329 rpcz_store.cc:276] 0709 13:02:06.544828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.546345 (+  1517us) service_pool.cc:224] Handling call
0709 13:02:06.546593 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:06.675921 (+129328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:06.675956 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:06.676009 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:06.676054 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.616107 (+940053us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400815954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:07.616177 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.616338 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296846208}
I20260709 13:02:07.599255 10853 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:07.706482  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.663972  9324 rpcz_store.cc:276] 0709 13:02:06.633894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.638361 (+  4467us) service_pool.cc:224] Handling call
0709 13:02:07.610099 (+971738us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:07.641305 (+ 31206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25154304}
I20260709 13:02:07.756659  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.757341  8726 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:07.758019  8575 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:07.749733  8577 raft_consensus.cc:2376] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3. Re-sending same reply.
I20260709 13:02:07.755254  9297 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:07.723767  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:07.727273  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.760167  8728 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:07.706912  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:07.723127  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:07.762049  8658 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:07.708634  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:07.724406  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:07.704397  8657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2179) took 1153 ms (client timeout 1504 ms). Trace:
I20260709 13:02:07.713215  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 634) took 1027 ms. Trace:
I20260709 13:02:07.763384  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:07.709553  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:07.677871  8651 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:07.765092  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.760710  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:07.766146  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.765202  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:07.763176  8657 rpcz_store.cc:269] 0709 13:02:06.550906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.551060 (+   154us) service_pool.cc:224] Handling call
0709 13:02:07.704163 (+1153103us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:07.704295 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23293184}
I20260709 13:02:07.747877  8582 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:07.767382  8580 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:07.711567  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2050) took 1058 ms. Trace:
I20260709 13:02:07.778761  8731 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:07.775924  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:07.769529  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.767565  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:07.770262  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.789575  8654 rpcz_store.cc:276] 0709 13:02:06.653167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.968806 (+315639us) service_pool.cc:224] Handling call
0709 13:02:06.969148 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:07.134417 (+165269us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.134449 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:07.134494 (+    45us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:07.134531 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.711277 (+576746us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540018f1d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:07.711323 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:07.711489 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":372888320}
I20260709 13:02:07.791643 10835 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:02:07.776574  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 67122.7us
I20260709 13:02:07.792260  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.793040  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:07.794513  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:07.795385 10835 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:07.795868 10835 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.796308 10835 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:07.796487  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:07.797381 10895 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:07.793694  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.797832 10895 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.798174  8582 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:07.798245 10895 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:07.798641 10831 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:07.706050  9327 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:07.800202  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:07.807720  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 52541.2us
I20260709 13:02:07.800684  8576 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:07.800194  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:07.763625  9331 rpcz_store.cc:276] 0709 13:02:06.686253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.695549 (+  9296us) service_pool.cc:224] Handling call
0709 13:02:07.712955 (+1017406us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b54006900d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:07.713142 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1736130304}
I20260709 13:02:07.804776  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:07.800575  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.678857 10603 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.810405  8727 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:07.811335 10525 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:07.703826 10811 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:07.812799  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.679112 10819 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:07.820797  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.755723 10798 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:07.813532 10525 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:07.825899 10525 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:07.811952  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:07.814189  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:07.814941  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:07.755606 10893 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:07.820520  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:07.822786  8575 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:07.830319  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2145) took 1037 ms. Trace:
I20260709 13:02:07.759945 10857 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:07.832831  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:07.769649 10820 raft_consensus.cc:2764] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:02:07.809705 10791 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:07.878064  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:07.885309  8658 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:07.889714  8730 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:02:07.835237  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:07.890451  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:07.891036  9331 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:07.830816  8583 rpcz_store.cc:276] 0709 13:02:06.793037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.793230 (+   193us) service_pool.cc:224] Handling call
0709 13:02:07.820344 (+1027114us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:07.830232 (+  9888us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27463040}
I20260709 13:02:07.761752  8653 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:07.819428 10851 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:07.761610 10854 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260709 13:02:07.819839 10879 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:07.889744  8622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.517s (SENT) [suppressed 32 similar messages]
I20260709 13:02:07.899745  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:07.879345 10617 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:07.821484 10646 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:07.898012  9329 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:02:07.943460  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53721.3us
I20260709 13:02:07.879113 10870 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:07.802178 10831 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.944993  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:07.829378 10828 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:38080 (request call id 82) took 2747 ms. Trace:
I20260709 13:02:07.957201  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:07.819414 10769 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260709 13:02:07.836059  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:07.966691  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 131481us
I20260709 13:02:07.809244 10812 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260709 13:02:07.895047 10791 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:07.965732  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 155628us
I20260709 13:02:07.902595 10789 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:07.818933 10800 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
I20260709 13:02:07.896083 10620 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:07.947834  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:07.997922  8727 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:08.000519  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 55575.2us
I20260709 13:02:07.944056 10857 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:08.007017  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:08.005116  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 192975us
I20260709 13:02:07.998452  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:07.956990 10646 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.825214 10811 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:07.964195 10902 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:08.044168 10819 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:07.996135  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 631) took 1412 ms. Trace:
I20260709 13:02:08.016578  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:08.066046  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:08.182502  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 164857us
I20260709 13:02:08.073446  8730 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:08.184038  8731 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:08.062201  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:08.097314 10525 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.014752 10791 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:08.102375  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 95452.1us
I20260709 13:02:07.959126 10870 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.180552 10795 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:08.007614 10872 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:07.958295 10879 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.188228 10525 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.203299 10525 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.067898 10646 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.203740 10525 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.017623 10603 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:08.003142 10820 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.185s
W20260709 13:02:08.194187  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 177662us
I20260709 13:02:08.202131 10642 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:08.210053 10642 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.210422 10642 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.015389  9324 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
I20260709 13:02:08.213078 10525 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.194319  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:08.198944  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 200526us
I20260709 13:02:08.061218 10652 raft_consensus.cc:993] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:02:07.902370 10893 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.066742  8725 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:08.251507  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:08.181560  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 381277us
W20260709 13:02:08.190888  9322 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.523s	user 0.010s	sys 0.001s
I20260709 13:02:08.200165 10795 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.192126 10878 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.181602 10604 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.194279 10870 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.206000 10809 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:08.043641 10886 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:08.272445  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.186084 10904 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:08.251624  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:08.063689 10831 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:08.252704  8730 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.066208 10857 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.186290 10902 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.193113 10796 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:08.053349 10910 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.173s
I20260709 13:02:08.197265 10867 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.187604  9327 rpcz_store.cc:276] 0709 13:02:06.554472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.561204 (+  6732us) service_pool.cc:224] Handling call
0709 13:02:06.687827 (+126623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:07.806549 (+1118722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.942774 (+136225us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400702bd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:07.966611 (+ 23837us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2340255104}
I20260709 13:02:08.253651  9322 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:08.333472  8731 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.251420 10603 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.051723 10620 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.216096 10872 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.275341 10525 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.351867 10525 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.261639 10795 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:08.258301 10836 raft_consensus.cc:1081] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Signalling peer 453e644bad5d410ab9be85b353e28dc2 to start an election
I20260709 13:02:08.328197  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.355829  8575 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:08.328850  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.357595  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:08.266923 10604 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:08.328994  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64235us
W20260709 13:02:08.329757  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61210.7us
I20260709 13:02:08.258291 10652 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:08.257851 10671 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.584s	user 0.010s	sys 0.000s
I20260709 13:02:08.374439  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:08.327946 10913 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.337126 10900 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.338011  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.338340 10886 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:08.339581  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.278972 10911 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260709 13:02:08.334384 10796 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.184984 10798 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.328438 10857 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.343743 10902 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:08.344059  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.358628  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:08.430269  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71733us
I20260709 13:02:08.259563 10875 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.431164  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:08.345654 10915 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:08.380156 10525 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.380712  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:08.434656  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54001.3us
I20260709 13:02:08.436211  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:08.384061 10811 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.273543 10878 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.353667 10603 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.384891 10652 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.403649  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.404217  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.410181  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:08.449426  8583 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:08.415928  8580 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.616s	user 0.008s	sys 0.000s
I20260709 13:02:08.350858 10867 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.390777 10769 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:08.455430  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:08.420804 10905 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:02:08.456575  9325 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:08.450402  8580 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:08.470703  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.451058  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:08.449811  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:08.403157 10900 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.439276 10525 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:08.474367  8729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.014s	sys 0.000s
I20260709 13:02:08.625530  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:08.449157 10854 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:08.580435 10888 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:08.575163  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:08.574252 10920 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:02:08.388490 10913 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.446782 10814 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:02:08.625908  8729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.015s	sys 0.000s
I20260709 13:02:08.490744  8583 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:08.626523  8730 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:08.430773 10918 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:08.627842  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.007499 10828 rpcz_store.cc:276] 0709 13:02:05.072470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.072688 (+   218us) service_pool.cc:224] Handling call
0709 13:02:05.180061 (+107373us) log.cc:844] Serialized 487 byte log entry
0709 13:02:05.801024 (+620963us) spinlock_profiling.cc:243] Waited 7.71 ms on lock 0x7b1800278da0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b29522 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:07.803948 (+2002924us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13390,"raft.queue_time_us":67330,"raft.run_cpu_time_us":9366,"raft.run_wall_time_us":140941,"spinlock_wait_cycles":17738112,"thread_start_us":57321,"threads_started":2,"wal-append.queue_time_us":97844,"wal-append.run_cpu_time_us":2157,"wal-append.run_wall_time_us":1107653}
I20260709 13:02:08.546555 10910 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:08.479535 10769 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:08.642964 10925 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.627569 10800 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:08.644672  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.631137  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:08.649102  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.651010  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:08.666662  8724 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:08.666913  8658 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:08.625797  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.656109  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:08.678334  8730 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.656798  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:08.658077  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.658397  8548 leader_election.cc:400] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4.
I20260709 13:02:08.632643 10879 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:08.626915 10809 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.643538  8731 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:08.630877 10854 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.659688 10914 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.647117 10671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41978 (request call id 82) took 3914 ms. Trace:
I20260709 13:02:08.730729  8548 leader_election.cc:403] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:08.709734  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:08.717003  8730 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
W20260709 13:02:08.722661  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 286422us
I20260709 13:02:08.650776  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 628) took 2096 ms. Trace:
I20260709 13:02:08.651852 10918 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:08.721984 10835 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.925s	user 0.012s	sys 0.000s
I20260709 13:02:08.457217 10915 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.745133  8583 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:08.862104  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1680) took 1013 ms. Trace:
I20260709 13:02:08.862776  8727 rpcz_store.cc:276] 0709 13:02:07.848978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.795507 (+946529us) service_pool.cc:224] Handling call
0709 13:02:08.861343 (+ 65836us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.861629 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.861669 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.861727 (+    58us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:08.861789 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.861874 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.862024 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:08.866189 10926 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
W20260709 13:02:08.867550 10646 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.617s	user 0.008s	sys 0.004s
I20260709 13:02:08.873152 10646 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.875401 10646 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.875973  8481 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 301) took 3216 ms. Trace:
W20260709 13:02:08.877911 10928 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 13:02:08.878769  8479 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 305) took 2694 ms. Trace:
W20260709 13:02:08.879202 10642 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.668s	user 0.008s	sys 0.004s
I20260709 13:02:08.879613 10642 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.880736  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2947646ebfc34bd7a5a55312f5812ee2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:02:08.881812 10603 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.885792  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.879212  8479 rpcz_store.cc:276] 0709 13:02:06.184320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.189419 (+  5099us) service_pool.cc:224] Handling call
0709 13:02:08.878700 (+2689281us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:07.388267 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:07.526516 (+138249us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:07.526547 (+    31us) write_op.cc:435] Acquired schema lock
0709 13:02:07.526570 (+    23us) tablet.cc:662] Decoding operations
0709 13:02:07.526927 (+   357us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:07.526978 (+    51us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:07.527010 (+    32us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:02:07.527189 (+   179us) tablet.cc:701] Row locks acquired
0709 13:02:07.527206 (+    17us) write_op.cc:260] PREPARE: finished
0709 13:02:07.527342 (+   136us) write_op.cc:270] Start()
0709 13:02:07.527474 (+   132us) write_op.cc:276] Timestamp: P: 1783602127527293 usec, L: 0
0709 13:02:07.527495 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:02:07.528218 (+   723us) log.cc:844] Serialized 535 byte log entry
0709 13:02:07.947435 (+419217us) op_driver.cc:464] REPLICATION: finished
0709 13:02:08.012927 (+ 65492us) write_op.cc:301] APPLY: starting
0709 13:02:08.043060 (+ 30133us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:08.068729 (+ 25669us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:08.186162 (+117433us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:08.212000 (+ 25838us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:08.260995 (+ 48995us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:02:08.261063 (+    68us) write_op.cc:312] APPLY: finished
0709 13:02:08.264850 (+  3787us) log.cc:844] Serialized 23 byte log entry
0709 13:02:08.265053 (+   203us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:08.265156 (+   103us) write_op.cc:454] Released schema lock
0709 13:02:08.265211 (+    55us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:08.265577 (+   366us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:08.265643 (+    66us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":610010,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":11630,"apply.run_cpu_time_us":2253,"apply.run_wall_time_us":257379,"num_ops":1,"prepare.queue_time_us":845245,"prepare.run_cpu_time_us":5971,"prepare.run_wall_time_us":150016,"replication_time_us":381591}]]}
I20260709 13:02:08.867981  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 467) took 1058 ms. Trace:
I20260709 13:02:08.887220  8656 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:08.870007  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:08.888482  8727 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.774343  8731 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:08.877060 10646 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:08.876363  8481 rpcz_store.cc:276] 0709 13:02:05.660079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.660264 (+   185us) service_pool.cc:224] Handling call
0709 13:02:08.875884 (+3215620us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:06.371121 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:06.533850 (+162729us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:06.535048 (+  1198us) write_op.cc:435] Acquired schema lock
0709 13:02:06.541349 (+  6301us) tablet.cc:662] Decoding operations
0709 13:02:06.637348 (+ 95999us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:06.650194 (+ 12846us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:06.670727 (+ 20533us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:02:06.696281 (+ 25554us) tablet.cc:701] Row locks acquired
0709 13:02:06.702320 (+  6039us) write_op.cc:260] PREPARE: finished
0709 13:02:06.724133 (+ 21813us) write_op.cc:270] Start()
0709 13:02:06.737110 (+ 12977us) write_op.cc:276] Timestamp: P: 1783602126724052 usec, L: 0
0709 13:02:06.746040 (+  8930us) op_driver.cc:348] REPLICATION: starting
0709 13:02:06.782080 (+ 36040us) log.cc:844] Serialized 535 byte log entry
0709 13:02:07.652492 (+870412us) op_driver.cc:464] REPLICATION: finished
0709 13:02:07.751945 (+ 99453us) write_op.cc:301] APPLY: starting
0709 13:02:07.752033 (+    88us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:07.752131 (+    98us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:07.752269 (+   138us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:07.752514 (+   245us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:07.752696 (+   182us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:02:07.752735 (+    39us) write_op.cc:312] APPLY: finished
0709 13:02:07.752953 (+   218us) log.cc:844] Serialized 23 byte log entry
0709 13:02:07.753138 (+   185us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:07.753211 (+    73us) write_op.cc:454] Released schema lock
0709 13:02:07.753264 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:07.806777 (+ 53513us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:07.811776 (+  4999us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":99203,"apply.run_cpu_time_us":1683,"apply.run_wall_time_us":1689,"num_ops":1,"prepare.queue_time_us":233585,"prepare.run_cpu_time_us":16038,"prepare.run_wall_time_us":1009066,"raft.queue_time_us":337980,"raft.run_cpu_time_us":4392,"raft.run_wall_time_us":275564,"replication_time_us":906300,"thread_start_us":843255,"threads_started":4,"wal-append.queue_time_us":613895}]]}
I20260709 13:02:08.898106  8623 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:08.909113  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:08.910820  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:08.912047 10525 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:08.913980  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 279929us
I20260709 13:02:08.887135  8730 rpcz_store.cc:276] 0709 13:02:07.809225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.655738 (+846513us) service_pool.cc:224] Handling call
0709 13:02:08.867920 (+212182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:08.923338  8580 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:08.679085 10814 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.720927 10828 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:08.946599  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.793668 10878 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:08.925197  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:08.959224  8658 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:08.820092 10604 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.792757 10796 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:08.965758  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:08.904867 10931 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:02:08.933207  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:08.871922 10835 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.776525 10872 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:08.791522 10924 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.980209  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.722565 10900 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.983006  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.984297  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.924818  9323 raft_consensus.cc:2955] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:08.871238 10857 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:08.680639 10800 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.654639 10888 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.948642  8548 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:08.985375  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.991411  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.953770 10525 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:08.930004  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:08.915055 10652 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:08.741677 10854 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.924927 10825 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4.
I20260709 13:02:08.954841  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:09.020957  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66741.2us
I20260709 13:02:08.957706  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.023705 10525 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:08.979044 10924 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.966574  8580 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:09.030368  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.030973  8730 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:08.887072 10769 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:08.945137 10603 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:08.869091  9325 rpcz_store.cc:276] 0709 13:02:06.553515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.560749 (+  7234us) service_pool.cc:224] Handling call
0709 13:02:08.389966 (+1829217us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:08.429233 (+ 39267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.475178 (+ 45945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.645277 (+170099us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6609792}
I20260709 13:02:08.970392 10878 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:08.972318  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.047545  8583 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:09.048795  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:08.973057  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.050209 10915 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.050328  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.051818  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.052842  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:09.053146 10652 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:09.053977  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:09.055569 10912 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:09.055943  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:09.056126 10912 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:09.056666  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1593) took 1040 ms. Trace:
I20260709 13:02:09.057063  8577 rpcz_store.cc:276] 0709 13:02:08.016594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.016767 (+   173us) service_pool.cc:224] Handling call
0709 13:02:08.039257 (+ 22490us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.190447 (+151190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.190486 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.190540 (+    54us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:08.190587 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.056417 (+865830us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54002d7dd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:09.056463 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.056581 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":347016064}
I20260709 13:02:09.058038 10835 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.058281 10620 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.058590  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:08.973577 10921 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:08.974396 10932 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.967679 10642 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:08.928115 10646 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:08.977550  8655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2166) took 2704 ms (client timeout 2110 ms). Trace:
I20260709 13:02:08.730829 10671 rpcz_store.cc:276] 0709 13:02:04.761006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.770142 (+  9136us) service_pool.cc:224] Handling call
0709 13:02:04.828719 (+ 58577us) log.cc:844] Serialized 487 byte log entry
0709 13:02:05.032110 (+203391us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b1800047ce0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409b6fb0b 00007f1409bf4b63 00007f1409c1c752 00007f1409c149fa
0709 13:02:08.634115 (+3602005us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":148781,"raft.queue_time_us":154990,"raft.run_cpu_time_us":1393,"raft.run_wall_time_us":1830,"spinlock_wait_cycles":14066560,"thread_start_us":51845,"threads_started":1,"wal-append.queue_time_us":49336,"wal-append.run_cpu_time_us":1675,"wal-append.run_wall_time_us":1002796}
I20260709 13:02:08.948235 10875 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:08.971452 10895 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.173s	user 0.007s	sys 0.006s
I20260709 13:02:08.859876 10918 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.772984 10809 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.032459  8581 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:09.033229 10825 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:02:09.034062  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.034548  8575 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:09.035800  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.042066  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1668) took 1318 ms. Trace:
I20260709 13:02:09.042382  8580 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:09.029603  9327 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:02:09.143786  8729 rpcz_store.cc:276] 0709 13:02:07.723729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.724552 (+   823us) service_pool.cc:224] Handling call
0709 13:02:07.724915 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.759102 (+ 34187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.759137 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:07.813197 (+ 54060us) log.cc:844] Serialized 26 byte log entry
0709 13:02:08.649156 (+835959us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:08.649227 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.649346 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.041812 (+392466us) raft_consensus.cc:1685] finished
0709 13:02:09.041833 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.041962 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2687,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":14947,"thread_start_us":689525,"threads_started":1,"wal-append.queue_time_us":763120}
I20260709 13:02:09.050415  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.053809 10652 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.986173 10842 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:08.762790 10925 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.061357  8732 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:09.061683  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:08.976238 10908 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.795s	user 0.003s	sys 0.000s
I20260709 13:02:09.074822 10924 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.077667 10932 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:08.782384 10914 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:09.036556 10853 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.277s	user 0.003s	sys 0.010s
I20260709 13:02:09.060014 10620 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:08.958384 10814 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.092311  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:09.214125  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 121891us
I20260709 13:02:09.214881  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:09.081681  8655 rpcz_store.cc:269] 0709 13:02:06.273104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.273828 (+   724us) service_pool.cc:224] Handling call
0709 13:02:08.977321 (+2703493us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540005a0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:08.977456 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2474080128}
I20260709 13:02:08.965320 10828 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.148276  8727 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:09.092549 10835 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:09.157262  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 95688.5us
I20260709 13:02:09.123037 10646 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.123355 10642 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:09.166527  8583 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:09.081660 10921 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:09.169250  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 258432us
I20260709 13:02:09.234648  8584 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:02:09.171341 10525 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:09.173358 10283 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:09.217692  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:09.227799  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:09.230909  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.141777 10895 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.247627  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.248777  8655 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:09.250370  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:09.253741  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 262268us
I20260709 13:02:09.212389 10924 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:09.217355 10671 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:09.057085 10912 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.222419 10875 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:09.272289  8577 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:09.240096 10620 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:09.256847  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:09.166966 10842 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.262476  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:09.286429  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1571) took 2432 ms. Trace:
I20260709 13:02:09.295994  8579 rpcz_store.cc:276] 0709 13:02:06.851030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.317761 (+466731us) service_pool.cc:224] Handling call
0709 13:02:07.318073 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.323468 (+  5395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.323514 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:07.334285 (+ 10771us) log.cc:844] Serialized 26 byte log entry
0709 13:02:07.704780 (+370495us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b1800052960. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:02:07.770223 (+ 65443us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:07.770292 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.770395 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.975655 (+1205260us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:09.283112 (+307457us) raft_consensus.cc:1685] finished
0709 13:02:09.283134 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.283282 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5028,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":8461,"spinlock_wait_cycles":16866048,"thread_start_us":65240,"threads_started":1,"wal-append.queue_time_us":616939}
W20260709 13:02:09.298655  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 298) took 2646 ms (client timeout 1551 ms). Trace:
W20260709 13:02:09.299403  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71693.8us
I20260709 13:02:09.299916  8650 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:09.301270  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:09.253082 10915 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:09.299247  8659 rpcz_store.cc:269] 0709 13:02:06.652684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.758995 (+106311us) service_pool.cc:224] Handling call
0709 13:02:09.298389 (+2539394us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b540018fbd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:09.298564 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1452492032}
I20260709 13:02:09.277379  8732 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:09.307889  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:09.266130 10820 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.999s	user 0.035s	sys 0.053s
I20260709 13:02:09.233737 10835 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.308776  8655 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 2: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:09.269035 10525 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:09.317308 10525 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.320147  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.321025  8657 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:09.291929 10842 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.255633 10800 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.253510 10934 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:02:09.324604  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67012.4us
I20260709 13:02:09.263288 10935 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:09.246296 10642 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:09.331971  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 114323us
I20260709 13:02:09.332741  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:09.334043  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:09.330686  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.339279  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:09.286554  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 237571us
I20260709 13:02:09.344424  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.287113  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:09.322522 10820 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.038s	sys 0.053s
W20260709 13:02:09.346947  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 90170.1us
I20260709 13:02:09.366232 10809 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.350998 10825 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 13:02:09.366357  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2055) took 2706 ms. Trace:
I20260709 13:02:09.367312 10620 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.367952  8651 rpcz_store.cc:276] 0709 13:02:06.660147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.047735 (+387588us) service_pool.cc:224] Handling call
0709 13:02:07.054201 (+  6466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:07.785761 (+731560us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.366049 (+1580288us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b540013a454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:09.366238 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1433201536}
I20260709 13:02:09.379017 10642 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.345995  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:09.382540 10828 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.283830 10671 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.331557 10900 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:09.326412 10888 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.284979 10924 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.326632 10875 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:09.332063 10934 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.406805  9327 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:09.319518 10915 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.429176  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1676) took 1626 ms. Trace:
I20260709 13:02:09.349583 10935 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.590530  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:09.593605  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2188) took 1134 ms. Trace:
I20260709 13:02:09.597041  8731 rpcz_store.cc:276] 0709 13:02:07.802860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.767070 (+964210us) service_pool.cc:224] Handling call
0709 13:02:08.773903 (+  6833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.774197 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:08.954552 (+180355us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.428776 (+474224us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.428832 (+    56us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:09.428889 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.428957 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.429102 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":71731,"raft.run_wall_time_us":16441,"spinlock_wait_cycles":43155712,"thread_start_us":473722,"threads_started":2,"wal-append.queue_time_us":309823}
I20260709 13:02:09.596189  8655 rpcz_store.cc:276] 0709 13:02:08.458929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.327136 (+868207us) service_pool.cc:224] Handling call
0709 13:02:09.592950 (+265814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.593235 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.593264 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.593306 (+    42us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:09.593347 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.593424 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.593549 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:09.602145 10652 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:09.612625  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:09.614078 10950 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.635537 10950 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.637807  9324 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:09.638881  9324 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
 from {username='slave'} at 127.0.0.1:43664
I20260709 13:02:09.639570  9324 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:02:09.641084  9324 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:02:09.645789 10923 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.103s	user 0.006s	sys 0.008s
I20260709 13:02:09.647442  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 484) took 1888 ms. Trace:
I20260709 13:02:09.648559  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.420876 10890 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:09.475296 10888 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.616849 10946 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.617640  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:09.538651 10941 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.231s	user 0.002s	sys 0.000s
I20260709 13:02:09.624185  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:09.624886  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.627102 10525 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.628695  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:09.666694  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:09.629209  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:09.631451  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.634486  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 303) took 2370 ms. Trace:
I20260709 13:02:09.640427  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.640583 10950 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.406050 10671 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.475517 10617 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:09.539729 10835 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.384806 10944 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:09.581208 10815 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:09.647890  9331 rpcz_store.cc:276] 0709 13:02:07.759520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.004595 (+245075us) service_pool.cc:224] Handling call
0709 13:02:08.048491 (+ 43896us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.052641 (+  4150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.052693 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.182291 (+129598us) log.cc:844] Serialized 26 byte log entry
0709 13:02:08.351859 (+169568us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:09.363436 (+1011577us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b18002262a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:09.375375 (+ 11939us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b18002262a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf
0709 13:02:09.375521 (+   146us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.375750 (+   229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.376260 (+   510us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b5400519354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:09.376315 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.629343 (+253028us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b24000c53a8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:02:09.647209 (+ 17866us) raft_consensus.cc:1685] finished
0709 13:02:09.647232 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.647377 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":74640,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":11678,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":264,"spinlock_wait_cycles":49928704,"thread_start_us":203608,"threads_started":2,"wal-append.queue_time_us":224223}
I20260709 13:02:09.615064 10935 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.658486 10914 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.660712 10915 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:09.541651 10854 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.506s	user 0.000s	sys 0.009s
W20260709 13:02:09.537528 10791 consensus_meta.cc:287] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.325s	user 0.012s	sys 0.002s
W20260709 13:02:09.508378 10795 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.132s	user 0.007s	sys 0.005s
I20260709 13:02:09.432811 10798 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:09.663542  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:09.664755 10646 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.431651 10875 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.430771 10911 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:09.433887 10900 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:09.507831 10828 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.431559 10934 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:09.670321  8652 rpcz_store.cc:276] 0709 13:02:07.264264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.331478 (+ 67214us) service_pool.cc:224] Handling call
0709 13:02:07.356326 (+ 24848us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:07.590828 (+234502us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.634221 (+2043393us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54000c85d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:09.634398 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":492425472}
I20260709 13:02:09.509480  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:09.679272  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55176.2us
I20260709 13:02:09.591511  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:09.654124  8575 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:09.581882 10924 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.562896 10853 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.476423 10814 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.648136  8730 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:09.735821  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.538312 10895 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.407416 10932 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.692124  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.692690  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.696009  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:09.594530 10816 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.003s	sys 0.012s
I20260709 13:02:09.474817 10925 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.697650  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2182) took 1248 ms. Trace:
I20260709 13:02:09.698045  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.748131  8656 rpcz_store.cc:276] 0709 13:02:08.444445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.444583 (+   138us) service_pool.cc:224] Handling call
0709 13:02:08.444920 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.887671 (+442751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.686255 (+798584us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b5400380254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:09.692815 (+  6560us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1016205696}
I20260709 13:02:09.748818  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:09.749441  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.750038  8656 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:09.750653 10853 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.751052  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.751442  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.752223  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.752640  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.752812 10853 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:09.753487 10816 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.004s	sys 0.012s
W20260709 13:02:09.699662  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47108.9us
I20260709 13:02:09.755151  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:09.755296  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2187) took 1296 ms. Trace:
I20260709 13:02:09.755892  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.539577 10921 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.708464  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:09.759413  8583 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:09.678593 10774 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:09.757203  9297 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:09.755707  8654 rpcz_store.cc:276] 0709 13:02:08.458681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.472538 (+ 13857us) service_pool.cc:224] Handling call
0709 13:02:09.754646 (+1282108us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:09.754816 (+   170us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.754990 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.755014 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.755054 (+    40us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:09.755097 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.755149 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.755241 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15207808}
I20260709 13:02:09.762918  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.686174 10950 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:09.685971 10944 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:09.776088 10867 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.675140 10888 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.717664  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.777709  8652 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:09.697199 10854 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.696969 10791 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.736933  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:09.784358  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47505.3us
I20260709 13:02:09.661487 10890 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.793040  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.722752 10911 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:09.747781 10922 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.093s	user 0.000s	sys 0.002s
I20260709 13:02:09.756494  8655 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:09.713792 10875 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:09.710721 10646 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.758615  8732 raft_consensus.cc:2410] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:09.761137  8579 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:09.698601 10792 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:09.763782 10283 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:09.753260  8656 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:09.819728  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.820844  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.688174 10617 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.822005 10791 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:09.823020 10617 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:09.765939  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 69998.6us
I20260709 13:02:09.742513  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 431) took 1293 ms. Trace:
I20260709 13:02:09.769238 10925 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:09.770048  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:09.770933 10842 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:02:09.771296 10870 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.504s	user 0.010s	sys 0.003s
I20260709 13:02:09.757529 10671 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260709 13:02:09.750185  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:09.772589 10913 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:09.775104 10937 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.392s	user 0.002s	sys 0.020s
I20260709 13:02:09.698277 10795 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.846585 10842 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:09.859872 10795 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:09.860704  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 463) took 2095 ms. Trace:
I20260709 13:02:09.785087  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:09.860595 10842 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.861032  8466 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9e8a9411169f407cba1b44443edc9840 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:02:09.861552  8651 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 13:02:09.861634 10842 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:09.861785  8466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 181511us
I20260709 13:02:09.699056 10815 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.758283  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:09.680562 10946 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.817572  9329 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:09.765453 10950 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:09.724632 10828 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.846495  8575 rpcz_store.cc:276] 0709 13:02:08.449493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.741708 (+1292215us) service_pool.cc:224] Handling call
0709 13:02:09.741998 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.742173 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.742198 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.742239 (+    41us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:09.742280 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.742331 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.742451 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:09.830088  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.887096  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.849079  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.754129 10900 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.824064  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:09.814558 10873 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:09.855696  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:09.854193 10870 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:09.815995  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:09.857379  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:09.885255  9324 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.885210  9325 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:09.776649 10944 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:09.861097  8724 rpcz_store.cc:276] 0709 13:02:07.770535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.667434 (+896899us) service_pool.cc:224] Handling call
0709 13:02:08.667788 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.668226 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.668268 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.766888 (+ 98620us) log.cc:844] Serialized 26 byte log entry
0709 13:02:09.173084 (+406196us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:09.173142 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.174588 (+  1446us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.774437 (+599849us) spinlock_profiling.cc:243] Waited 9.44 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:09.860430 (+ 85993us) raft_consensus.cc:1685] finished
0709 13:02:09.860449 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.860645 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1665,"raft.queue_time_us":10240,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":46826,"spinlock_wait_cycles":21711232,"thread_start_us":279103,"threads_started":1,"wal-append.queue_time_us":513724}
I20260709 13:02:09.885485 10950 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.887437  8656 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:10.099509  8654 raft_consensus.cc:2376] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3. Re-sending same reply.
W20260709 13:02:10.067963  8584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1598) took 1580 ms (client timeout 1963 ms). Trace:
I20260709 13:02:09.887761  8575 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:10.101558  8579 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:10.102588  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.102707  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.104697  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1599) took 1429 ms. Trace:
W20260709 13:02:10.105840  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 349489us
I20260709 13:02:10.101292 10890 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:10.106864  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 251162us
W20260709 13:02:10.108239  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.122998  8575 rpcz_store.cc:276] 0709 13:02:08.675013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.103603 (+1428590us) service_pool.cc:224] Handling call
0709 13:02:10.103874 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.104047 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.104070 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:10.104218 (+   148us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:10.104377 (+   159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.104465 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.104627 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.105767  8583 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:09.895469  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110429us
W20260709 13:02:10.100829  8584 rpcz_store.cc:269] 0709 13:02:08.450983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.648803 (+197820us) service_pool.cc:224] Handling call
0709 13:02:10.030960 (+1382157us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54003a3c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:10.031093 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1305736960}
I20260709 13:02:10.117192  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2164) took 2517 ms. Trace:
W20260709 13:02:10.095090 10949 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.704s	user 0.007s	sys 0.008s
W20260709 13:02:10.083411 10942 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.700s	user 0.018s	sys 0.007s
I20260709 13:02:10.128675  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:10.128868  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:10.129415  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1596) took 1704 ms. Trace:
I20260709 13:02:10.103643  8579 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:10.130148  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:10.126261 10795 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:10.129138  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.131973 10814 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:09.853605 10825 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:09.883464 10925 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:10.103376 10918 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.950s	user 0.015s	sys 0.000s
I20260709 13:02:10.134153  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:10.136715  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:10.143448  8575 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:09.883445  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2180) took 1432 ms. Trace:
I20260709 13:02:10.149748  8653 rpcz_store.cc:276] 0709 13:02:08.443610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.446168 (+  2558us) service_pool.cc:224] Handling call
0709 13:02:08.626817 (+180649us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.659541 (+ 32724us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.659583 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.659638 (+    55us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:08.659698 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.848103 (+1188405us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b54000591d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:09.850131 (+  2028us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.875475 (+ 25344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64178432}
I20260709 13:02:09.916646 10617 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:10.110370 10932 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:10.151087 10617 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:10.134128  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:10.129794  8580 rpcz_store.cc:276] 0709 13:02:08.425264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.160489 (+735225us) service_pool.cc:224] Handling call
0709 13:02:09.160914 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.217726 (+ 56812us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.217780 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:09.226419 (+  8639us) log.cc:844] Serialized 26 byte log entry
0709 13:02:09.309442 (+ 83023us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:09.309528 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.309711 (+   183us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.084456 (+774745us) spinlock_profiling.cc:243] Waited 8.36 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:10.129109 (+ 44653us) raft_consensus.cc:1685] finished
0709 13:02:10.129129 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.129331 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56315,"raft.queue_time_us":56450,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":18373,"spinlock_wait_cycles":19218816,"thread_start_us":40479,"threads_started":1,"wal-append.queue_time_us":135380}
I20260709 13:02:10.128299  8576 rpcz_store.cc:276] 0709 13:02:07.599922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.809499 (+209577us) service_pool.cc:224] Handling call
0709 13:02:07.809834 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.188192 (+378358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.188240 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.202675 (+ 14435us) log.cc:844] Serialized 26 byte log entry
0709 13:02:08.359421 (+156746us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:08.359477 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.359568 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.738862 (+1379294us) spinlock_profiling.cc:243] Waited 7.53 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:09.928720 (+189858us) raft_consensus.cc:1685] finished
0709 13:02:09.928751 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.117110 (+188359us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53153,"raft.queue_time_us":154583,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":43865,"spinlock_wait_cycles":17310976,"thread_start_us":462414,"threads_started":2,"wal-append.queue_time_us":206743}
I20260709 13:02:10.114218 10867 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:10.153885 10867 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:10.155158 10867 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:10.118652  9323 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.131s	user 0.012s	sys 0.009s
W20260709 13:02:10.139133 10620 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.018s	sys 0.033s
I20260709 13:02:10.117337 10854 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:10.158921  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:10.177289  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.120359 10646 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:10.139922 10811 raft_consensus.cc:2764] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 13:02:10.140755 10814 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:10.130066 10889 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:10.163051 10652 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:10.143374 10795 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:10.226064 10947 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:10.168413 10825 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:10.268808  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1684) took 2262 ms. Trace:
I20260709 13:02:10.432222  8729 rpcz_store.cc:276] 0709 13:02:08.006297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.166761 (+1160464us) service_pool.cc:224] Handling call
0709 13:02:09.205774 (+ 39013us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.207309 (+  1535us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.207338 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.207379 (+    41us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:09.207423 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.268353 (+1060930us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b54001e5fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:10.268401 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.268541 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3043712}
W20260709 13:02:10.434302  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 275443us
I20260709 13:02:10.436249  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:10.350432  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1606) took 1409 ms. Trace:
I20260709 13:02:10.342497  8584 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260709 13:02:10.429893  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.055s	sys 0.068s
I20260709 13:02:10.342015  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2062) took 3012 ms. Trace:
I20260709 13:02:10.437597  8577 rpcz_store.cc:276] 0709 13:02:08.941303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.058306 (+117003us) service_pool.cc:224] Handling call
0709 13:02:10.341313 (+1283007us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54002d73d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:10.350311 (+  8998us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489540480}
W20260709 13:02:10.438091  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.055s	sys 0.068s
I20260709 13:02:10.249387 10814 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:10.454152  8698 connection.cc:256] RPC call timeout handler was delayed by -1.0255153489999884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
I20260709 13:02:10.441773  8650 rpcz_store.cc:276] 0709 13:02:07.329929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.371523 (+ 41594us) service_pool.cc:224] Handling call
0709 13:02:09.299512 (+1927989us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.300425 (+   913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.341778 (+1041353us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b540018fbd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:10.341922 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4430924544}
W20260709 13:02:10.454797  8698 connection.cc:256] RPC call timeout handler was delayed by -1.0066153490000147s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
I20260709 13:02:10.455801  8696 leader_election.cc:400] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:10.261415 10815 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:10.298249 10652 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:10.456158 10816 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.049s	sys 0.082s
I20260709 13:02:10.298162 10828 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:10.159082 10918 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:10.454079 10795 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:10.453756 10888 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:10.458878 10921 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:10.480343  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 263190us
I20260709 13:02:10.460063 10889 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:10.474016  9327 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:10.463167 10947 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:10.478299 10816 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.049s	sys 0.082s
I20260709 13:02:10.484923  8733 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:10.496600  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:10.508160  8728 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:10.510800  8658 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:10.511065  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 460) took 2747 ms (client timeout 3107 ms). Trace:
I20260709 13:02:10.511759  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2036) took 2764 ms. Trace:
I20260709 13:02:10.512297  8733 rpcz_store.cc:276] 0709 13:02:07.742900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.769737 (+ 26837us) service_pool.cc:224] Handling call
0709 13:02:10.484441 (+2714704us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:10.484666 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.507557 (+ 22891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.507773 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7652224}
W20260709 13:02:10.511601  8728 rpcz_store.cc:269] 0709 13:02:07.764117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.496260 (+2732143us) service_pool.cc:224] Handling call
0709 13:02:10.510996 (+ 14736us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.488775  9324 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:10.520152  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:10.509959 10870 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:10.500792 10828 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:10.483225 10652 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:10.525825 10961 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.309s	user 0.000s	sys 0.001s
I20260709 13:02:10.526564  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:43664 (request call id 491) took 1066 ms. Trace:
I20260709 13:02:10.526741 10964 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:10.515290 10947 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:10.531586  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 417) took 3127 ms. Trace:
I20260709 13:02:10.532164 10911 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:10.535691  8480 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:10.506577 10889 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:10.522339 10917 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:10.537807  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 418) took 3133 ms. Trace:
I20260709 13:02:10.488464 10925 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:10.539619 10959 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:02:10.540055  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:10.539857 10875 consensus_meta.cc:287] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.712s	user 0.008s	sys 0.002s
W20260709 13:02:10.563941  8731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.015s	sys 0.005s
I20260709 13:02:10.520881 10946 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:10.566376  8581 rpcz_store.cc:276] 0709 13:02:07.404248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.797498 (+393250us) service_pool.cc:224] Handling call
0709 13:02:07.806944 (+  9446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.134291 (+1327347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.531255 (+1396964us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000b5c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:10.531505 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2816887040}
I20260709 13:02:10.544745 10918 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:10.568326  8582 rpcz_store.cc:276] 0709 13:02:07.404555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.798926 (+394371us) service_pool.cc:224] Handling call
0709 13:02:07.799243 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.012499 (+213256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.012533 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:08.012574 (+    41us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:08.012608 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.537530 (+2524922us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b540017a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:10.537583 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.537738 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489728896}
W20260709 13:02:10.578843  8731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.016s	sys 0.005s
I20260709 13:02:10.581878  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:10.583333  8655 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:10.585647  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:10.569248  8480 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 3, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:10.589759  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2040) took 1255 ms. Trace:
I20260709 13:02:10.589176  8480 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 3, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:10.590364  8731 rpcz_store.cc:276] 0709 13:02:09.334206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.614711 (+280505us) service_pool.cc:224] Handling call
0709 13:02:09.615050 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.615307 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:09.716863 (+101556us) log.cc:844] Serialized 14 byte log entry
0709 13:02:10.589277 (+872414us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:10.589346 (+    69us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:10.589436 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.589529 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.589671 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7614,"prepare.queue_time_us":87033,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"thread_start_us":965251,"threads_started":2}
I20260709 13:02:10.587683  8655 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:10.593582  8480 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:10.595340  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:10.596014  8655 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:10.570329  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 657) took 2295 ms. Trace:
I20260709 13:02:10.598227  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:10.545791 10828 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:10.598773  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 72443.2us
I20260709 13:02:10.551679  9324 rpcz_store.cc:276] 0709 13:02:09.508165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.638748 (+130583us) service_pool.cc:224] Handling call
0709 13:02:10.522906 (+884158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.602067  8655 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:10.603736  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:10.594965 10825 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:10.593959 10879 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:10.584578 10875 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:10.600607  9323 rpcz_store.cc:276] 0709 13:02:08.268914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.271338 (+  2424us) service_pool.cc:224] Handling call
0709 13:02:08.387943 (+116605us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.429858 (+ 41915us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.431728 (+  1870us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:08.660712 (+228984us) log.cc:844] Serialized 487 byte log entry
0709 13:02:08.924656 (+263944us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:10.162351 (+1237695us) log.cc:844] Serialized 14 byte log entry
0709 13:02:10.516758 (+354407us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.520772 (+  4014us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.525324 (+  4552us) raft_consensus.cc:1685] finished
0709 13:02:10.525342 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.563787 (+ 38445us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24716,"raft.queue_time_us":52098,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":5514,"spinlock_wait_cycles":44815744,"thread_start_us":476798,"threads_started":2,"wal-append.queue_time_us":646801,"wal-append.run_cpu_time_us":2024,"wal-append.run_wall_time_us":1033387}
I20260709 13:02:10.615648 10875 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:10.609599 10946 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:10.616871  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:10.618103  8652 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:10.618304  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:10.605383 10798 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:10.618978 10798 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:10.619441 10889 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:10.619590  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:10.621865  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:10.622427  8724 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:10.622354  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:10.624128  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:10.628326  8731 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:10.630105  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:10.630762  8728 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:10.634171  8727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.022s	sys 0.019s
I20260709 13:02:10.584293 10964 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:10.634564  8727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.023s	sys 0.019s
I20260709 13:02:10.637969  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2069) took 1546 ms. Trace:
I20260709 13:02:10.638478  8654 rpcz_store.cc:276] 0709 13:02:09.091930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.113791 (+1021861us) service_pool.cc:224] Handling call
0709 13:02:10.114109 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.116179 (+  2070us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.116214 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:10.116723 (+   509us) log.cc:844] Serialized 26 byte log entry
0709 13:02:10.477172 (+360449us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:10.477228 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.477304 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.620470 (+143166us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:10.625963 (+  5493us) raft_consensus.cc:1685] finished
0709 13:02:10.625982 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.637882 (+ 11900us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1393,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":4547,"spinlock_wait_cycles":3532672,"thread_start_us":259475,"threads_started":1,"wal-append.queue_time_us":431189}
I20260709 13:02:10.640391  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:10.641040  8654 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:02:10.642349  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 309) took 1556 ms. Trace:
I20260709 13:02:10.643335  8731 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:10.642729  8659 rpcz_store.cc:276] 0709 13:02:09.086439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.376965 (+290526us) service_pool.cc:224] Handling call
0709 13:02:09.592901 (+215936us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.650750 (+ 57849us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.650781 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:09.662409 (+ 11628us) log.cc:844] Serialized 26 byte log entry
0709 13:02:09.703400 (+ 40991us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:09.703447 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.703554 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.642159 (+938605us) raft_consensus.cc:1685] finished
0709 13:02:10.642177 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.642290 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":57854,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":9526,"spinlock_wait_cycles":1792,"thread_start_us":7968,"threads_started":1,"wal-append.queue_time_us":480105}
I20260709 13:02:10.644230  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:10.644721  8625 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:10.646438 10964 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:10.647655 10828 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:10.648285  8726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1674) took 2890 ms (client timeout 2768 ms). Trace:
I20260709 13:02:10.648705 10881 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:10.608577 10652 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:10.638031 10921 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:10.653045  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.654065  8724 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:10.656073  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.719117  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.648860  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:10.659024  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:10.665678  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:10.674602  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:10.677997  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.681830  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.725805  8575 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
W20260709 13:02:10.693537  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.690138  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:10.657508  8726 rpcz_store.cc:269] 0709 13:02:07.757779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.648053 (+2890274us) service_pool.cc:211] Skipping call since client already timed out
0709 13:02:10.648185 (+   132us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:02:10.726987  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.704077  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:10.705531  9294 connection.cc:256] RPC call timeout handler was delayed by -1.4639139679999857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
I20260709 13:02:10.712532  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.652937  8728 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:10.658695 10934 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.936s	user 0.008s	sys 0.008s
I20260709 13:02:10.777732 10934 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:10.780139 10934 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:10.706090 10825 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:10.727733  8698 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:10.781471 10825 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:10.782029  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54367.2us
I20260709 13:02:10.783835 10971 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:10.784135  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:10.785192 10946 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:10.787253  8659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:35192
I20260709 13:02:10.822701  8659 raft_consensus.cc:606] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:10.823236  8659 raft_consensus.cc:614] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:10.787895  8576 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:10.788956  9294 connection.cc:256] RPC call timeout handler was delayed by -1.2154616770000075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20260709 13:02:10.790350  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.776619  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:10.825989  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:10.882982  9323 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:10.883630  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57719.5us
I20260709 13:02:10.883870 10911 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:10.884680  9323 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:10.786049  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:10.781509 10652 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:10.885665  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 493) took 1029 ms. Trace:
I20260709 13:02:10.885893  8654 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:10.783859 10825 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:10.657150 10918 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:10.884580 10911 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:10.646484 10947 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:10.888671  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:10.717494  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:10.712395 10964 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:10.903256 10853 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:10.904922 10918 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:10.909703  9324 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:10.906956 10652 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:10.904996 10964 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.025601 10839 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:10.886093  9323 rpcz_store.cc:276] 0709 13:02:09.856585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.884084 (+1027499us) service_pool.cc:224] Handling call
0709 13:02:10.884453 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.885311 (+   858us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.885600 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:11.034544 10969 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.427s	user 0.000s	sys 0.001s
I20260709 13:02:11.039410 10911 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:10.908129 10888 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20260709 13:02:11.081058  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2194) took 2207 ms. Trace:
W20260709 13:02:11.084568  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 300271us
I20260709 13:02:11.095124  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:11.098577  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 177038us
W20260709 13:02:11.066587 10950 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.968s	user 0.013s	sys 0.002s
I20260709 13:02:11.101416  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.088294  8657 rpcz_store.cc:276] 0709 13:02:08.873660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.335411 (+461751us) service_pool.cc:224] Handling call
0709 13:02:09.336054 (+   643us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.336259 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:09.336543 (+   284us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.045061 (+1708518us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b180033ca00. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a
0709 13:02:11.080447 (+ 35386us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:11.080536 (+    89us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:11.080624 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.080733 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.080935 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21412,"prepare.queue_time_us":140,"prepare.run_cpu_time_us":136,"prepare.run_wall_time_us":136,"raft.queue_time_us":26045,"spinlock_wait_cycles":11896192,"thread_start_us":322091,"threads_started":1,"wal-append.queue_time_us":318264,"wal-append.run_cpu_time_us":1438,"wal-append.run_wall_time_us":1058859}
I20260709 13:02:11.060865 10652 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:11.065191 10964 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:11.102051 10912 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.764s	user 0.014s	sys 0.015s
I20260709 13:02:11.085237 10792 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:11.065802 10946 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:11.106909 10950 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.111943 10888 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:11.130143 10876 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:11.137621 10912 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.777s	user 0.015s	sys 0.015s
W20260709 13:02:11.148046  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.137877 10888 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.140731 10944 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.130985 10946 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:11.148749 10938 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.768s	user 0.036s	sys 0.022s
I20260709 13:02:11.168802  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2033) took 3426 ms. Trace:
W20260709 13:02:11.169910 10974 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 13:02:11.171291  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 471) took 1835 ms. Trace:
I20260709 13:02:11.171720  8727 rpcz_store.cc:276] 0709 13:02:09.335921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.763100 (+427179us) service_pool.cc:224] Handling call
0709 13:02:09.763385 (+   285us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.767323 (+  3938us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.767352 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:09.767820 (+   468us) log.cc:844] Serialized 26 byte log entry
0709 13:02:10.634965 (+867145us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:10.634993 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.635046 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:11.171050 (+536004us) raft_consensus.cc:1685] finished
0709 13:02:11.171073 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.171220 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14386,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":26061,"thread_start_us":740723,"threads_started":1,"wal-append.queue_time_us":1044338}
I20260709 13:02:11.153506 10888 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.147523 10652 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.182410 10652 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.141083 10918 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.187711 10946 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.188624 10932 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.189352 10932 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.190699 10950 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.165733 10944 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:11.194083 10814 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.730s	user 0.010s	sys 0.000s
I20260709 13:02:11.195760 10944 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.196677  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:11.197731 10798 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.578s	user 0.006s	sys 0.004s
I20260709 13:02:11.198124  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.201558 10925 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:11.202493 10934 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:11.203228 10934 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:11.129686 10853 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:11.206300 10853 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.208630  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 312) took 2121 ms. Trace:
W20260709 13:02:11.166867 10938 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.784s	user 0.036s	sys 0.022s
I20260709 13:02:11.209237  8651 rpcz_store.cc:276] 0709 13:02:09.087465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.376979 (+289514us) service_pool.cc:224] Handling call
0709 13:02:09.377393 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.862042 (+484649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.208267 (+1346225us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b540018f6d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:11.208542 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112561920}
I20260709 13:02:11.204414 10934 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.169497  8725 rpcz_store.cc:276] 0709 13:02:07.741965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.754277 (+ 12312us) service_pool.cc:224] Handling call
0709 13:02:08.066305 (+312028us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.253535 (+187230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.168426 (+2914891us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400339850. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.168698 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554855168}
I20260709 13:02:11.194552 10814 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:11.225647  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.219075 10925 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:11.239135  8650 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:11.243206  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 653) took 1474 ms. Trace:
I20260709 13:02:11.250005  8581 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:11.250730  8581 pending_rounds.cc:85] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 5
I20260709 13:02:11.217682 10798 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.251159  8581 pending_rounds.cc:107] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:11.253834  8581 raft_consensus.cc:2889] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:11.253429 10950 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.261906 10950 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.262310  8577 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:11.262944 10814 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.264272 10950 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.264276 10911 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:11.265127 10925 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.265859 10913 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:11.266263 10913 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:11.269476  9323 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:11.267786 10911 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:11.279350  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.285775  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:11.280248  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.287690  8728 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:11.267073 10913 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.254783 10985 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:11.274854  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:11.294275  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:11.273839 10857 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:11.299613 10857 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.301986 10980 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:11.302542 10980 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:11.302953 10980 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.303359 10888 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.305382 10946 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:11.302196 10798 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.308771 10857 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.310752  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 638) took 3923 ms. Trace:
I20260709 13:02:11.310575  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.312057  8581 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.313761  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.314390  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.315819  8657 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:11.314404  8655 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:11.311269  9328 rpcz_store.cc:276] 0709 13:02:07.387330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.387713 (+   383us) service_pool.cc:224] Handling call
0709 13:02:09.635563 (+2247850us) spinlock_profiling.cc:243] Waited 6.46 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:09.635745 (+   182us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.757797 (+122052us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.757828 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:09.758312 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 13:02:10.136667 (+378355us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:10.136727 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.310445 (+1173718us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b5400517a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:11.310503 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:11.310553 (+    50us) raft_consensus.cc:1685] finished
0709 13:02:11.310563 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.310692 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":193,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":5685,"spinlock_wait_cycles":294601344,"thread_start_us":323445,"threads_started":1,"wal-append.queue_time_us":354370,"wal-append.run_cpu_time_us":1947,"wal-append.run_wall_time_us":1001935}
I20260709 13:02:11.323724  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.324766  8655 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:11.296646  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:11.326663  8725 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:11.326494  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:11.319052 10888 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:11.332504  8576 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:11.334118 10857 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:11.340279 10964 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.346814  8581 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:11.350275  8653 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:11.353639 10950 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.354290 10950 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.353750  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.351866  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.359464  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.362035  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.363355  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:11.365010 10864 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.365675 10853 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:11.365783 10864 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.367188 10986 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:11.367662 10986 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.368046 10986 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.366379 10853 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:11.370813 10940 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:11.370261 10853 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.357976 10988 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:11.372787 10617 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:11.373090 10988 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.370251 10990 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.374228 10990 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.374626 10990 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.375316  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2168) took 1637 ms. Trace:
I20260709 13:02:11.375873 10988 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.375701  8584 rpcz_store.cc:276] 0709 13:02:09.738218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.132019 (+393801us) service_pool.cc:224] Handling call
0709 13:02:10.342227 (+210208us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.454196 (+111969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.375078 (+920882us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54002d73d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.375243 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":482511616}
I20260709 13:02:11.343817 10913 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.361302 10950 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.363298 10944 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.384702 10947 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.417641 10980 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.427415 10980 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:11.421466 10944 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.428661 10944 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.385886 10864 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.431824 10864 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.432723 10798 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:11.432487 10864 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.433373 10798 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.435590 10864 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.410307 10916 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:11.436475 10916 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:11.421569 10964 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.437757 10964 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:11.437434 10916 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.438248 10964 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.438735 10916 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.423967 10698 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.441295 10964 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:11.441788 10698 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.442857  8578 raft_consensus.cc:1813] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Handling vote request from an unknown peer d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:11.443490  8578 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:11.444795  8725 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:11.460988  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:11.462108  9329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 664) took 3105 ms (client timeout 3183 ms). Trace:
I20260709 13:02:11.462885 10990 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:11.466017  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2192) took 2770 ms (client timeout 2002 ms). Trace:
W20260709 13:02:11.466459  8658 rpcz_store.cc:269] 0709 13:02:08.695669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.050156 (+354487us) service_pool.cc:224] Handling call
0709 13:02:11.465718 (+2415562us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540025c454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.465930 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3126147456}
W20260709 13:02:11.462675  9329 rpcz_store.cc:269] 0709 13:02:08.356427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.395038 (+ 38611us) service_pool.cc:224] Handling call
0709 13:02:11.461880 (+3066842us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54007dcdd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.462009 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3145602304}
W20260709 13:02:11.471426  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 108105us
I20260709 13:02:11.424657 10947 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.435726 10798 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:11.448480  9325 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 674) took 2043 ms (client timeout 1639 ms). Trace:
I20260709 13:02:11.454066 10642 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:11.455958  8652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2197) took 1676 ms (client timeout 1501 ms). Trace:
W20260709 13:02:11.489668  8652 rpcz_store.cc:269] 0709 13:02:09.779798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.783723 (+  3925us) service_pool.cc:224] Handling call
0709 13:02:11.455686 (+1671963us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b5400139054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.455864 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1893512448}
I20260709 13:02:11.453240  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:11.492379  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.472437  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.494786 10646 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:11.527305 10944 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:11.494203  9325 rpcz_store.cc:269] 0709 13:02:09.404937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.419873 (+ 14936us) service_pool.cc:224] Handling call
0709 13:02:11.448186 (+2028313us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400630bd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.448358 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321808128}
I20260709 13:02:11.493964 10839 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.532804 10798 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:11.557790 10996 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 13:02:11.526593 10864 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:11.524922 10875 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:11.525950 10918 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.525557 10642 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.499356  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:11.629254  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 129995us
I20260709 13:02:11.617650 10913 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.558545  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 652) took 1843 ms. Trace:
W20260709 13:02:11.511607  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.578475  8656 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:11.562040  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 675) took 2086 ms. Trace:
W20260709 13:02:11.610592  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 122846us
W20260709 13:02:11.619436  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.596369 10976 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:11.317487 10985 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.617333 10646 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:11.633328  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 650) took 1960 ms. Trace:
I20260709 13:02:11.584219 10932 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.637450  9331 rpcz_store.cc:276] 0709 13:02:09.672971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.786298 (+113327us) service_pool.cc:224] Handling call
0709 13:02:09.786733 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.829776 (+ 43043us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.829815 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.829886 (+    71us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:09.829950 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.633109 (+1803159us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b54003b01d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:11.633150 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.633275 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98231168}
I20260709 13:02:11.635182  9330 rpcz_store.cc:276] 0709 13:02:09.475356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.665816 (+190460us) service_pool.cc:224] Handling call
0709 13:02:09.694969 (+ 29153us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.781938 (+ 86969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.781985 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.782044 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:09.782109 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.561710 (+1779601us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b54008f0cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:11.561773 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.561959 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199284608}
I20260709 13:02:11.639828  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.627339 10918 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.625784 10875 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.589838 10986 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.621855 10839 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.632472 10913 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:11.688750  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48943.3us
I20260709 13:02:11.700266 10698 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.700587 10947 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.700682  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.702028 10944 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.702656 10944 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.703022 10944 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:11.703421 10944 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.703226 10940 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:11.704010 10940 consensus_queue.cc:1243] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:02:11.704713 10950 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:11.705070 10950 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:11.625368 10864 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.706035 10950 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.636057 10990 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.692499  8576 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:11.692195 10646 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.697480  8579 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:11.626287 10798 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:11.714097  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 79602.2us
I20260709 13:02:11.640201 10934 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.675446 10875 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:11.645458 10918 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.643661 10999 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260709 13:02:11.706888 10642 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.634111  9327 rpcz_store.cc:276] 0709 13:02:09.715085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.859368 (+144283us) service_pool.cc:224] Handling call
0709 13:02:09.859693 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.509749 (+650056us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.558082 (+1048333us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b5400702bd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:11.558351 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320250112}
I20260709 13:02:11.727825 10934 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.741936 10934 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.742275 10864 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.707738 11001 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:11.744663 11001 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.745074 11001 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:11.745700 10839 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.713960 10988 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.642364 10976 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.715788 10913 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.752992  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:11.757015  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55152.2us
I20260709 13:02:11.758360  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:11.760426 10839 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:11.760828 10839 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.761200 10839 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.761511  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.764533  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.765373  8575 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:11.765820  8726 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:11.767575  8575 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:11.767994  8726 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:11.769018  8583 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:11.770699  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.770951  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.771502  8575 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:11.772080  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.773180  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.773914  8575 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:11.779295 10792 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.779480  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.780184  8580 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:11.779968 10792 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.782236  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.782884  8582 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:11.783016  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.784037  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.781122  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.784808  8580 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:11.762905 10916 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 13:02:11.788549  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.762576 10925 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:11.790359  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:11.794176 10985 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.794710 10976 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.794807  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.795539  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.796211  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.796305 10617 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.797233  8728 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:02:11.798439 10888 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:11.801819  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.836530  8582 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.802055 10947 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:11.806990  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.827560  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:11.833388  8655 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:11.833545  8658 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:11.826908  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.833110  8659 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:11.833026  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.834789  8652 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:11.808636  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.838192  8724 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:11.839982  8657 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:11.841003  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.754179 10988 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.762614 10890 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 13:02:11.843266  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.844348  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.845933  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.846900  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.855623  8726 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:11.847791  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.849251  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.850425  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.857506  8725 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 13:02:11.781116  9322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 661) took 3452 ms (client timeout 2768 ms). Trace:
I20260709 13:02:11.852273  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:11.837857  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:11.852967  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.858489  8724 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:11.862500  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:11.863682  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.769619 10853 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:11.867731 10853 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.853185 10947 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.870657 10853 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.861042  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.832435 10873 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:11.865163 11006 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:11.865756  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.866446 10988 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.879846 10873 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.889950  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.890447  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.891216  8725 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:11.890905 10839 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.892431  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.887041 10911 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.866389  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:11.853443 10888 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.867122  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:11.860414 10798 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.897246  8652 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:11.901944 10792 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.902678 10918 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.904471  8658 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:11.899469 10932 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.902683 10792 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:11.842347 10617 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.905937 10792 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:11.906963 10792 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.907544 10986 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.907805 10617 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.852484 10917 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
W20260709 13:02:11.908936  8578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1578) took 4837 ms (client timeout 1517 ms). Trace:
I20260709 13:02:11.872383  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.893388 10839 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.910635  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.911780  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.912508 10283 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.912756  8583 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:11.912916 10283 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:11.913446 10283 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.914434  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:11.913270  8582 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:11.863722  8730 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:11.243842  9326 rpcz_store.cc:276] 0709 13:02:09.768155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.887719 (+1119564us) service_pool.cc:224] Handling call
0709 13:02:10.892884 (+  5165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.893257 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:11.151033 (+257776us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.241703 (+ 90670us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:11.241987 (+   284us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:11.242079 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.242197 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.242403 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":288400,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":1456,"thread_start_us":335797,"threads_started":2}
I20260709 13:02:11.835830 10642 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 13:02:11.859414 10652 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260709 13:02:11.854533 11001 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.918329 10642 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:11.918737 10642 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:11.919167  8576 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:11.875185 10921 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 13:02:11.919695 10642 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.861326 10964 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 13:02:11.880373  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:11.861905  9322 rpcz_store.cc:269] 0709 13:02:08.328944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.343751 (+ 14807us) service_pool.cc:224] Handling call
0709 13:02:11.780887 (+3437136us) spinlock_profiling.cc:243] Waited 2.86 s on lock 0x7b5400693a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.781035 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6580992256}
I20260709 13:02:11.853339 10980 raft_consensus.cc:2764] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:02:11.874837 10976 raft_consensus.cc:2764] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:02:11.860791  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:11.898836 10977 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.768s	user 0.007s	sys 0.000s
I20260709 13:02:11.920086 11001 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:11.909416  8578 rpcz_store.cc:269] 0709 13:02:07.071292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.076860 (+  5568us) service_pool.cc:224] Handling call
0709 13:02:11.908696 (+4831836us) spinlock_profiling.cc:243] Waited 4.35 s on lock 0x7b5400179b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:11.908835 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10011696512}
I20260709 13:02:11.922456  8652 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:11.880728  8655 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:11.922742  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.950618  8575 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 1.
I20260709 13:02:11.951978  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.953843 11017 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:11.954493 11017 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:11.955804 11017 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.958216  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.916489 10889 raft_consensus.cc:2764] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:02:11.940081  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.942430  8724 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:11.924372 10798 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:11.959283  8578 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:11.905015 10918 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:11.914078 10913 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.978361  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.979110  8578 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:02:11.979532 10913 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.980887  8733 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:11.960846  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:11.929091 10642 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:11.983325 10283 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.966914 11019 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:11.966979  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 52584.6us
I20260709 13:02:12.194545  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:11.899302 10888 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.904709  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 494) took 1817 ms. Trace:
I20260709 13:02:12.195459  9323 rpcz_store.cc:276] 0709 13:02:10.087407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.099480 (+1012073us) service_pool.cc:224] Handling call
0709 13:02:11.124455 (+ 24975us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.297745 (+173290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.904440 (+606695us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54006900d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:11.904642 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297282944}
I20260709 13:02:11.974575  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.198309  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.199221  8725 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:12.200392  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:11.975510  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.201761  8582 raft_consensus.cc:1813] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Handling vote request from an unknown peer d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:12.202540  8582 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:12.203627  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:11.975651 10934 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:12.208771 10934 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:11.950011 10857 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:12.209635 10857 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:12.210206 10857 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:11.981699  8575 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:11.925316  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45056.7us
I20260709 13:02:12.218089  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:12.220135  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:12.221453 10888 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:12.222038 10888 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:12.222671 10888 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:12.226212  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.972455 11017 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:12.092666  8578 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:12.111676 10698 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:02:12.113514  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.972091 10980 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:11.971210 10875 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:11.972967 10798 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:11.984736  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.154191 11019 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.152956 10964 consensus_queue.cc:1243] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:12.055845 10911 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:12.252007 10975 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:12.252746 10975 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:12.255434 10975 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:12.077703 10947 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:12.123764 10873 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:12.029035 10283 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:12.256879  8654 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:12.182801  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260709 13:02:12.257500  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.183271 11021 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:11.984035  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 125491us
I20260709 13:02:12.263667  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:12.186988  8623 connection.cc:256] RPC call timeout handler was delayed by -1.5225667069999531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
I20260709 13:02:12.190466 10944 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:11.970309 11018 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.205163  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.236179  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:12.238482  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:12.233741 10879 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:12.232195 10917 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:12.234057 10698 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:12.239912  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.241253 10875 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:12.243196  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:12.282131  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:12.281909  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:12.283735  8659 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:12.285144  8651 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:12.285573  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:12.286226  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:12.286958  8651 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:12.287578  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:12.244962 10991 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.783s	user 0.010s	sys 0.007s
I20260709 13:02:12.245970 10798 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:12.235491 10980 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:12.247740  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.247820  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.273023 10769 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:12.248410 10881 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:12.271606 11021 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:12.292362  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:12.292580 10881 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:12.293015 10881 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:12.293210  8653 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:11.971396 11020 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:12.293520  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.294061 11021 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:12.293813 11020 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.296350 10950 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:12.297648 10944 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:12.266374  8582 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:12.285593 10990 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 13:02:12.298875  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:12.299649  8582 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:12.272919 10938 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 13:02:12.300518  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:12.301057  8582 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:12.266374  8578 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:12.303612 10873 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:12.304339  8725 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:12.270361  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:12.270967  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:12.249728 10652 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:12.315613  8653 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:12.317806 10698 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.319056 10698 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:12.269968  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:12.272382  8575 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:12.261329 10913 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:12.272811 10988 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:12.305694  8730 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:12.268309  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:12.325671  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57420.5us
I20260709 13:02:12.312453  8578 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:12.274415 10879 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:12.283396 11014 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260709 13:02:12.332528  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.274837 10839 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:12.320852 10857 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.341490  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:12.344684  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.354683  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.313643 11001 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:12.272135 10964 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260709 13:02:12.305152 10980 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:12.358398  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 88522.2us
I20260709 13:02:12.366320  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:12.366504  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:12.369573  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 103719us
I20260709 13:02:12.371284  8659 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:12.375268  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:12.360985  9326 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:12.361264  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.392872  8575 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:12.393759  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.394151  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:12.394320  8575 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:12.360445  8576 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:12.274168 11018 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.376080 10811 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:12.395756  8576 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:12.377190 10642 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.396428 10642 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.397023  8576 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:12.398046  9329 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:12.398679  9327 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:12.398964  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:12.353953 10879 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:12.400822  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:12.401316  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:12.402006  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:12.403025  8576 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:12.362113 10839 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.404150  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.404829  8575 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:12.316354 10652 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:12.381350 10938 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.345350 10990 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:12.359064 10911 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:12.444483 10911 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:12.444814  8584 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:12.444993 10911 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:12.340677 10988 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:12.388800  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.361882 10917 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.487179  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:12.501206  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 213626us
I20260709 13:02:12.507043  8582 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:12.387715 10881 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:12.360752  8578 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:12.406196  8575 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:12.404847  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:12.407434  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:12.352213 10913 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:12.432288  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.387274  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:12.446355 10988 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:12.377996  8623 connection.cc:256] RPC call timeout handler was delayed by -1.3194327739999494s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
I20260709 13:02:12.552716  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.837869  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 457) took 1007 ms. Trace:
W20260709 13:02:12.884923  8698 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.501s (SENT)
I20260709 13:02:12.896024  8582 rpcz_store.cc:276] 0709 13:02:11.830183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.795232 (+965049us) service_pool.cc:224] Handling call
0709 13:02:12.837149 (+ 41917us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.837415 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.837454 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:12.837510 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:12.837561 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.837643 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.837792 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:12.896649  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:12.839569  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 433348us
I20260709 13:02:12.541147  8651 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
W20260709 13:02:12.800038  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 398645us
W20260709 13:02:12.897605  8623 connection.cc:256] RPC call timeout handler was delayed by -1.2930899539999814s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1562s for a response
W20260709 13:02:12.902667  8623 connection.cc:256] RPC call timeout handler was delayed by -1.2684873319999497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3183s for a response
W20260709 13:02:12.903333  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.903143  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:12.898077  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 180725us
W20260709 13:02:12.904574  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:12.897364  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.898310  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:12.907262  9331 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:12.900184  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.900869  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1627) took 1098 ms. Trace:
I20260709 13:02:12.910104  8728 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:12.907274  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 499782us
W20260709 13:02:12.898101  8698 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.696s (SENT) [suppressed 18 similar messages]
I20260709 13:02:12.900470  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:12.910252  8575 rpcz_store.cc:276] 0709 13:02:11.802238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.405692 (+603454us) service_pool.cc:224] Handling call
0709 13:02:12.405939 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.900551 (+494612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.900800 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.917309  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:12.911268  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.898743  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:12.939603  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:12.941599  8732 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:12.944725  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 458) took 1114 ms. Trace:
I20260709 13:02:12.945204  8578 rpcz_store.cc:276] 0709 13:02:11.830502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.943817 (+1113315us) service_pool.cc:224] Handling call
0709 13:02:12.944161 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.944365 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.944400 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:12.944452 (+    52us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:12.944501 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.944563 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.944666 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.948082  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:12.948797  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.950026  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:12.952255  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:12.952903  8652 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
W20260709 13:02:12.756563 11024 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.325s	user 0.000s	sys 0.001s
I20260709 13:02:12.970857  9330 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 1. Current term is 2. Ops: []
W20260709 13:02:12.972631  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 73964.4us
I20260709 13:02:12.973677  8581 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:12.513633 10857 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:12.974404  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:12.972627  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1628) took 1170 ms. Trace:
I20260709 13:02:12.480914 10990 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:12.977200  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:13.006801  8652 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:12.487985  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:12.768810 11018 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:12.449030 10938 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:12.949636 10888 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.726s	user 0.015s	sys 0.007s
I20260709 13:02:12.940686 10975 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:12.942675 10964 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.942814 11021 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:13.008311  8584 rpcz_store.cc:276] 0709 13:02:11.802493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.445700 (+643207us) service_pool.cc:224] Handling call
0709 13:02:12.445969 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.446156 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:12.773277 (+327121us) log.cc:844] Serialized 14 byte log entry
0709 13:02:12.972124 (+198847us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:12.972196 (+    72us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:12.972269 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.972372 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.972543 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58652,"raft.queue_time_us":59865,"thread_start_us":466381,"threads_started":2}
I20260709 13:02:13.011384  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:13.026114  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 81559.3us
I20260709 13:02:12.487372 10917 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:13.029750  8623 connection.cc:256] RPC call timeout handler was delayed by -1.1904667069999846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
I20260709 13:02:13.031342  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:13.031446  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57139.7us
I20260709 13:02:13.066866  8550 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:12.961818  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:13.030817  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:13.034461  8582 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:13.034883 11030 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:13.007333 10881 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:13.023319 10698 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:13.037557  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:13.043313 10938 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:13.103523  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 200516us
W20260709 13:02:13.067935  8623 connection.cc:256] RPC call timeout handler was delayed by -1.0527667069999893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
I20260709 13:02:13.072494  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:13.074045  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 460) took 1170 ms. Trace:
I20260709 13:02:12.523554 10652 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:13.075886  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:13.062165 10839 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:13.047640 10975 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:13.092235  8657 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:13.045028 10642 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:13.047394 10888 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:13.182847 10698 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:13.281929  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 383706us
I20260709 13:02:13.308111  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2065) took 1049 ms. Trace:
I20260709 13:02:13.308516  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:13.120119  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:13.309190  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:13.154138  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2094) took 1347 ms (client timeout 1757 ms). Trace:
I20260709 13:02:13.157212  8651 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:13.311431  8582 rpcz_store.cc:276] 0709 13:02:11.904081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.020074 (+1115993us) service_pool.cc:224] Handling call
0709 13:02:13.034278 (+ 14204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.073686 (+ 39408us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.073931 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.153677  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2202) took 3367 ms. Trace:
W20260709 13:02:13.324390  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.331755  8727 rpcz_store.cc:276] 0709 13:02:12.258892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.072136 (+813244us) service_pool.cc:224] Handling call
0709 13:02:13.308042 (+235906us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.322958  8652 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
W20260709 13:02:13.322751  8658 rpcz_store.cc:269] 0709 13:02:11.806537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.851595 (+ 45058us) service_pool.cc:224] Handling call
0709 13:02:13.153910 (+1302315us) spinlock_profiling.cc:243] Waited 7.24 ms on lock 0x7b540031cc54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:13.154075 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16644480}
W20260709 13:02:13.324860  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.334190  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:13.075877 11021 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:13.334805  8652 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:13.738226  9328 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:13.432956  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:02:13.557862  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 88162.2us
I20260709 13:02:13.360638  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:13.945243  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 455) took 2122 ms. Trace:
I20260709 13:02:13.945463  8584 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 13:02:13.932643  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.946396  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.337466  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 276238us
I20260709 13:02:13.945715  8576 rpcz_store.cc:276] 0709 13:02:11.822732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.508219 (+685487us) service_pool.cc:224] Handling call
0709 13:02:12.508535 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.901236 (+392701us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.901281 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:12.901906 (+   625us) log.cc:844] Serialized 26 byte log entry
0709 13:02:13.097336 (+195430us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:13.097450 (+   114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.097967 (+   517us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:13.942425 (+844458us) raft_consensus.cc:1685] finished
0709 13:02:13.942456 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.945179 (+  2723us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9139,"raft.queue_time_us":407792,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":7287,"thread_start_us":143412,"threads_started":1,"wal-append.queue_time_us":480173}
I20260709 13:02:13.316656 10938 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:13.919582  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 227165us
I20260709 13:02:13.366060  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:13.949450  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2211) took 1030 ms. Trace:
W20260709 13:02:13.949937  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 641276us
I20260709 13:02:13.949872  8576 rpcz_store.cc:276] 0709 13:02:12.919234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.948596 (+1029362us) service_pool.cc:224] Handling call
0709 13:02:13.948919 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.949113 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.949139 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:13.949180 (+    41us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:13.949215 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.949269 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.949381 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.950454  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:13.951829  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.000s	sys 0.013s
W20260709 13:02:13.952217  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.000s	sys 0.013s
W20260709 13:02:13.952571  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.331755  8650 rpcz_store.cc:276] 0709 13:02:09.786526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.455201 (+668675us) service_pool.cc:224] Handling call
0709 13:02:10.455518 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.242184 (+786666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.153346 (+1911162us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54001d15d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:13.153596 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1800992512}
I20260709 13:02:13.953390  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:13.954063  8659 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.947s	user 0.003s	sys 0.009s
W20260709 13:02:13.943389  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 876311us
W20260709 13:02:13.946972  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.949100 10938 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:13.568697  8657 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:13.953660  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:13.321055 10652 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:13.947669  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1734) took 1596 ms (client timeout 1954 ms). Trace:
W20260709 13:02:13.954463  8659 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.947s	user 0.003s	sys 0.009s
I20260709 13:02:13.961503  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:13.993773  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:13.997471  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2206) took 1761 ms. Trace:
I20260709 13:02:13.598619  8579 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:13.719403  8582 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:14.033872  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:14.046629  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 463) took 1797 ms. Trace:
I20260709 13:02:14.046926  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:13.960924  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:13.335052  9330 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:14.006451  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:13.997766  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:14.035772  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:13.102977 10816 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:14.048913  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 462) took 1811 ms. Trace:
I20260709 13:02:14.049105  8550 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.049288  8582 rpcz_store.cc:276] 0709 13:02:12.237816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.373195 (+1135379us) service_pool.cc:224] Handling call
0709 13:02:13.718931 (+345736us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.048678 (+329747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.048864 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.000008  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:13.073185 10964 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:13.360430  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:14.032514  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 79819.3us
I20260709 13:02:14.034004  8724 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:14.080857 11027 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.034124  8584 rpcz_store.cc:276] 0709 13:02:12.236497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.076380 (+839883us) service_pool.cc:224] Handling call
0709 13:02:13.945169 (+868789us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.960100 (+ 14931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.997165 (+ 37065us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b54002c9cd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:13.997362 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1996803456}
I20260709 13:02:13.306747 11035 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:13.951942  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:14.083616  8658 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:14.040102  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:14.006825 10911 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.561s	user 0.006s	sys 0.010s
I20260709 13:02:13.045517 11026 raft_consensus.cc:2764] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:02:13.120618 10769 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:13.345738 11030 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:13.990367 10811 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.889s	user 0.033s	sys 0.056s
W20260709 13:02:14.043541  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 90210us
W20260709 13:02:13.774478 10798 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.453s	user 0.020s	sys 0.003s
I20260709 13:02:13.302304  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:13.987200 10997 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Append to log took a long time: real 2.496s	user 0.020s	sys 0.030s
I20260709 13:02:13.965819 10698 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:14.045511  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 97137.5us
I20260709 13:02:14.047106  8579 rpcz_store.cc:276] 0709 13:02:12.249379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.597946 (+1348567us) service_pool.cc:224] Handling call
0709 13:02:13.598375 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.046297 (+447922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.046558 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.048311  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 503) took 1016 ms. Trace:
I20260709 13:02:14.049434  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:13.550310 11034 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.036334 10975 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:13.120826  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:14.050781  8729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2071) took 1790 ms (client timeout 2167 ms). Trace:
I20260709 13:02:14.051666  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 668) took 1213 ms. Trace:
I20260709 13:02:14.037128  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 340) took 1090 ms. Trace:
W20260709 13:02:14.033016  8727 rpcz_store.cc:269] 0709 13:02:12.351606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.360352 (+1008746us) service_pool.cc:224] Handling call
0709 13:02:13.947610 (+587258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.969709 11046 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:14.052349 10839 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:14.053382  8726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 493) took 1690 ms (client timeout 2083 ms). Trace:
I20260709 13:02:13.153007 10986 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.901s
I20260709 13:02:14.034976 11037 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:14.058385 10950 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.762s	user 0.017s	sys 0.000s
W20260709 13:02:14.058689 10980 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.675s	user 0.009s	sys 0.003s
I20260709 13:02:14.050945 10816 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:14.064831 10881 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.962s	user 0.007s	sys 0.002s
I20260709 13:02:14.066619 11017 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:14.070746  9325 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:14.072883 11020 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:14.049911 10917 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:14.073706 10792 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:13.152453 10990 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:13.979082 11049 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:14.080173  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:14.088138 10911 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.081633 11027 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.087075 11035 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.094837  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:14.703421  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 664) took 2201 ms. Trace:
I20260709 13:02:14.303714  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2204) took 2460 ms. Trace:
I20260709 13:02:14.171034  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 509) took 1049 ms. Trace:
I20260709 13:02:14.704000  9325 rpcz_store.cc:276] 0709 13:02:12.502150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.525014 (+ 22864us) service_pool.cc:224] Handling call
0709 13:02:14.070261 (+1545247us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:14.070500 (+   239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.703108 (+632608us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.703357 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46659456}
W20260709 13:02:14.106150  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.705679 11017 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:14.643927  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:14.712248 10811 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.616s	user 0.044s	sys 0.070s
I20260709 13:02:14.205224 10913 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:14.634334  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.600906 10938 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:14.635100  8658 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.036s
I20260709 13:02:14.603865  8480 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 201) took 8018 ms. Trace:
I20260709 13:02:14.721544 11017 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.130025  8724 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:14.723278  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:14.106933  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 335) took 1161 ms. Trace:
I20260709 13:02:14.724054  8584 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.725530  8658 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:14.726022  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2217) took 1351 ms. Trace:
W20260709 13:02:14.726250  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2066) took 2467 ms (client timeout 2142 ms). Trace:
I20260709 13:02:14.726321  8658 rpcz_store.cc:276] 0709 13:02:13.375073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.951664 (+576591us) service_pool.cc:224] Handling call
0709 13:02:14.725961 (+774297us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:14.726742  8724 rpcz_store.cc:269] 0709 13:02:12.259169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.993420 (+1734251us) service_pool.cc:224] Handling call
0709 13:02:14.726161 (+732741us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.589460 11016 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:14.147645  8698 leader_election.cc:336] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.002s (SENT)
I20260709 13:02:14.727483 11016 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:14.727627  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.051199 11043 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:14.728353 11016 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.087177 11053 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:02:14.191425 11048 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:02:14.728801 11043 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:14.203905  8581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1619) took 2403 ms (client timeout 2196 ms). Trace:
I20260709 13:02:14.712872 11049 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.193773 10964 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:14.705132  9324 rpcz_store.cc:276] 0709 13:02:13.121357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.310389 (+189032us) service_pool.cc:224] Handling call
0709 13:02:14.170150 (+859761us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.714560  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 447) took 3440 ms. Trace:
I20260709 13:02:14.716220 10917 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:14.713299 11046 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:14.716605  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 636310us
W20260709 13:02:14.101994  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2095) took 2295 ms (client timeout 1775 ms). Trace:
W20260709 13:02:14.717483  8726 rpcz_store.cc:269] 0709 13:02:12.362552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.006137 (+1643585us) service_pool.cc:224] Handling call
0709 13:02:14.053320 (+ 47183us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:14.281957  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58725.6us
I20260709 13:02:14.550001  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 658) took 2156 ms. Trace:
I20260709 13:02:14.718587 10990 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.716490  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:14.201890  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:14.642040  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:14.145857 11044 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:14.730212 11043 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:14.643736  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2055) took 2770 ms (client timeout 1824 ms). Trace:
W20260709 13:02:14.745046  8731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2072) took 2472 ms (client timeout 2176 ms). Trace:
I20260709 13:02:14.719686 11020 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:14.745620  8731 rpcz_store.cc:269] 0709 13:02:12.273336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.201523 (+1928187us) service_pool.cc:224] Handling call
0709 13:02:14.744957 (+543434us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.746130 11020 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.746892  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:14.744608  8577 rpcz_store.cc:276] 0709 13:02:11.274338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.274606 (+   268us) service_pool.cc:224] Handling call
0709 13:02:11.297346 (+ 22740us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.297843 (+   497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:11.297875 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:11.298344 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.393731 (+ 95387us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:11.393783 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.393884 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:13.965851 (+2571967us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:14.714166 (+748315us) raft_consensus.cc:1685] finished
0709 13:02:14.714198 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.714474 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13340,"raft.queue_time_us":21369,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7291,"spinlock_wait_cycles":54238464,"thread_start_us":95029,"threads_started":1,"wal-append.queue_time_us":171687}
W20260709 13:02:14.744626  8729 rpcz_store.cc:269] 0709 13:02:12.260228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.997511 (+1737283us) service_pool.cc:224] Handling call
0709 13:02:14.050711 (+ 53200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.748046  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:14.748373  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:14.748780 11020 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:14.729758 11053 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:14.749661 11053 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:14.212317 10916 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:14.750466 10916 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:14.750553  8581 rpcz_store.cc:269] 0709 13:02:11.800227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.293242 (+493015us) service_pool.cc:224] Handling call
0709 13:02:14.203583 (+1910341us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b5400168ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.203784 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1562106240}
I20260709 13:02:14.751600 10916 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.752070  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:14.752686 10916 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:14.752895 10917 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:14.719830 10816 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.104441 10769 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:14.519320  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1735) took 2161 ms (client timeout 2124 ms). Trace:
I20260709 13:02:14.754240 10769 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.720570  8655 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:14.172135 11026 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:14.721865  9330 rpcz_store.cc:276] 0709 13:02:13.032342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.308846 (+276504us) service_pool.cc:224] Handling call
0709 13:02:14.048245 (+739399us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.205478  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:14.206679 11021 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:14.635505 11054 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:14.207479  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2191) took 2434 ms. Trace:
I20260709 13:02:14.704319 10698 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.760972 10642 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.761065 10990 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.761518 10642 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:14.761863  9322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 682) took 2401 ms (client timeout 1838 ms). Trace:
W20260709 13:02:14.762310  9322 rpcz_store.cc:269] 0709 13:02:12.360032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.361773 (+  1741us) service_pool.cc:224] Handling call
0709 13:02:14.205218 (+1843445us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1e9c 00007f1406af1a5b 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:14.761783 (+556565us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4242176}
I20260709 13:02:14.763618 10990 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:14.763226 11054 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.765714 11054 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:14.724205  8659 rpcz_store.cc:276] 0709 13:02:12.945518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.947029 (+  1511us) service_pool.cc:224] Handling call
0709 13:02:12.947311 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.947529 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:14.034919 (+1087390us) log.cc:844] Serialized 14 byte log entry
0709 13:02:14.106447 (+ 71528us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.106524 (+    77us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.106594 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.106699 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.106859 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1005611,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":332,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":16508,"thread_start_us":1098405,"threads_started":2}
I20260709 13:02:14.766124  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:14.766819  8724 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:14.767200  8655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2099) took 2498 ms (client timeout 1684 ms). Trace:
I20260709 13:02:14.767774 11021 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:14.767868  8655 rpcz_store.cc:269] 0709 13:02:12.268424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.270164 (+  1740us) service_pool.cc:224] Handling call
0709 13:02:14.767032 (+2496868us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.767135 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5533750528}
I20260709 13:02:14.768906 10938 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:14.768576  8654 rpcz_store.cc:269] 0709 13:02:11.806791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.852044 (+ 45253us) service_pool.cc:224] Handling call
0709 13:02:14.088930 (+2236886us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b54001310d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.101901 (+ 12971us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":906892032}
W20260709 13:02:14.770027  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 53614.8us
I20260709 13:02:14.770169 10642 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:14.770314  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:14.771205  8654 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:14.311650 10888 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:14.767117  9328 rpcz_store.cc:276] 0709 13:02:12.393923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.394027 (+   104us) service_pool.cc:224] Handling call
0709 13:02:12.397967 (+  3940us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.944322 (+1546355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.549645 (+605323us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54002148d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:14.549918 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3080832384}
I20260709 13:02:14.754487  9331 rpcz_store.cc:276] 0709 13:02:12.845236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.946357 (+101121us) service_pool.cc:224] Handling call
0709 13:02:13.999800 (+1053443us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:14.051592 (+ 51792us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35449728}
I20260709 13:02:14.551432  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2068) took 5459 ms. Trace:
I20260709 13:02:14.745553 10964 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 13:02:14.772806 10980 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.773161 10964 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:14.729794  8578 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:14.642823  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:14.772148 10917 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.712496 10825 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.774633  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:14.775236 10825 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.776207  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:14.776535  8582 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 13:02:14.776621  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 628826us
W20260709 13:02:14.777470  9331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 506) took 1702 ms (client timeout 2046 ms). Trace:
I20260709 13:02:14.777935 10825 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:14.777849  9331 rpcz_store.cc:269] 0709 13:02:13.075087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.775969 (+1700882us) service_pool.cc:224] Handling call
0709 13:02:14.777056 (+  1087us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.722757 11017 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:14.778213  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:14.293540 10988 consensus_meta.cc:287] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.339s	user 0.009s	sys 0.003s
I20260709 13:02:14.778857  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:14.779610  9331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 510) took 1497 ms (client timeout 1919 ms). Trace:
I20260709 13:02:14.779228 10988 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.755259 11046 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:14.780691  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:14.781344  9328 raft_consensus.cc:1813] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Handling vote request from an unknown peer 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:14.781924  9328 raft_consensus.cc:2410] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:14.770922  8580 rpcz_store.cc:276] 0709 13:02:11.773596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.785390 (+ 11794us) service_pool.cc:224] Handling call
0709 13:02:11.785716 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.874008 (+ 88292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:11.874038 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:11.874497 (+   459us) log.cc:844] Serialized 26 byte log entry
0709 13:02:12.132899 (+258402us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:12.132977 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.206932 (+2073955us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b5400169ed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:14.207000 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.207069 (+    69us) raft_consensus.cc:1685] finished
0709 13:02:14.207089 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.207391 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":898,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":8534,"spinlock_wait_cycles":202139008,"thread_start_us":204680,"threads_started":1,"wal-append.queue_time_us":266320,"wal-append.run_cpu_time_us":1963,"wal-append.run_wall_time_us":1006935}
I20260709 13:02:14.748939  8577 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:14.773851  8578 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:14.780735  8576 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:14.766789  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:14.368279  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.783905  8659 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:14.774473 11049 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:14.788163  8582 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:14.211876  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 333) took 1268 ms. Trace:
W20260709 13:02:14.281193  8725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2061) took 2252 ms (client timeout 1645 ms). Trace:
I20260709 13:02:14.784560  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2213) took 1864 ms. Trace:
W20260709 13:02:14.640929 10879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.604s	user 0.008s	sys 0.016s
W20260709 13:02:14.767486  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2068) took 2507 ms (client timeout 3292 ms). Trace:
W20260709 13:02:14.814951  8724 rpcz_store.cc:269] 0709 13:02:12.259723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.765877 (+2506154us) service_pool.cc:224] Handling call
0709 13:02:14.767436 (+  1559us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.816530  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 476) took 2786 ms. Trace:
I20260709 13:02:14.816884  8724 rpcz_store.cc:276] 0709 13:02:12.030378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.815723 (+2785345us) service_pool.cc:224] Handling call
0709 13:02:14.816031 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.816208 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.816241 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.816280 (+    39us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:14.816325 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.816375 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.816478 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.818312  8724 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:14.818149  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.818950  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 477) took 2788 ms. Trace:
I20260709 13:02:14.819238  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2207) took 1503 ms. Trace:
I20260709 13:02:14.819329  8724 rpcz_store.cc:276] 0709 13:02:12.030631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.817905 (+2787274us) service_pool.cc:224] Handling call
0709 13:02:14.818153 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.818733 (+   580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.818899 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.819892  8659 rpcz_store.cc:276] 0709 13:02:13.315718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.817805 (+1502087us) service_pool.cc:224] Handling call
0709 13:02:14.819163 (+  1358us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.821060  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 481) took 2789 ms. Trace:
I20260709 13:02:14.782764  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:14.804973  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.821622  8580 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:14.791643  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2213) took 1418 ms. Trace:
I20260709 13:02:14.791889  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:14.821973  8654 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:14.822322  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.641644  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:14.822527  8582 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:14.823201  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 175896us
W20260709 13:02:14.823690  8622 connection.cc:256] RPC call timeout handler was delayed by -2.4156937679999828s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1998s for a response
I20260709 13:02:14.823850  8582 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:14.782389  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 504) took 1733 ms. Trace:
W20260709 13:02:14.792610  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64913.3us
W20260709 13:02:14.824993  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60386.9us
I20260709 13:02:14.735085 10811 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:14.774376  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:14.826720  9297 leader_election.cc:336] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 3.107s (SENT)
I20260709 13:02:14.827019 11046 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:14.828028  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.826719  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:14.824658  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44862.6us
I20260709 13:02:14.828754 11046 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:14.829339  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:14.829607 11061 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:14.830154 11061 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:14.830536 11061 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.830458  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:14.831068  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.831871 11062 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:14.832445 11062 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:14.749481 11020 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:14.833633 11062 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.834444 11020 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:14.834734 11021 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:14.835055 11020 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:14.779971  9331 rpcz_store.cc:269] 0709 13:02:13.282315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.778625 (+1496310us) service_pool.cc:224] Handling call
0709 13:02:14.779556 (+   931us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.825603  9328 rpcz_store.cc:276] 0709 13:02:13.049020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.780425 (+1731405us) service_pool.cc:224] Handling call
0709 13:02:14.782329 (+  1904us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:14.548018  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1727) took 2206 ms (client timeout 2195 ms). Trace:
I20260709 13:02:14.825213  8582 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:02:14.836310  8728 rpcz_store.cc:269] 0709 13:02:12.341519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.547650 (+2206131us) service_pool.cc:211] Skipping call since client already timed out
0709 13:02:14.547849 (+   199us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:02:14.837875  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 482) took 2806 ms. Trace:
I20260709 13:02:14.640038  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2185) took 2870 ms. Trace:
I20260709 13:02:14.795729  8651 rpcz_store.cc:276] 0709 13:02:12.944824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.947493 (+  2669us) service_pool.cc:224] Handling call
0709 13:02:13.156955 (+209462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.332235 (+175280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.211674 (+879439us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b540031cc50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.211816 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":480857856}
I20260709 13:02:14.771909  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:14.838456 10811 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:14.839382  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 67539.2us
I20260709 13:02:14.839200 11061 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.839385  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.772493  8656 rpcz_store.cc:276] 0709 13:02:09.091615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.110543 (+1018928us) service_pool.cc:224] Handling call
0709 13:02:11.468106 (+1357563us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.635183 (+167077us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.900907 (+1265724us) spinlock_profiling.cc:243] Waited 7.46 ms on lock 0x7b54003ab220. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c19a02 00007f1409c15dd6 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:02:13.946157 (+1045250us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b540025c454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:14.550998 (+604841us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540025c450. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.551310 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3390593152}
I20260709 13:02:14.738622  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:14.840608  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:14.754041  8732 rpcz_store.cc:269] 0709 13:02:12.358151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.518986 (+2160835us) service_pool.cc:211] Skipping call since client already timed out
0709 13:02:14.519211 (+   225us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:02:14.840934 10811 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:14.769256  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.796669  8731 raft_consensus.cc:2410] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:02:14.733150  8584 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:14.745311  8733 rpcz_store.cc:269] 0709 13:02:11.872891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.878681 (+  5790us) service_pool.cc:224] Handling call
0709 13:02:14.553661 (+2674980us) spinlock_profiling.cc:243] Waited 90 ms on lock 0x7b5400338954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.643623 (+ 89962us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":206894848}
I20260709 13:02:14.841009  8727 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:14.786432  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:14.703953  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 667) took 1906 ms. Trace:
W20260709 13:02:14.793177  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.804612  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:14.842275  8731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 480) took 2811 ms (client timeout 3093 ms). Trace:
I20260709 13:02:14.842521  8579 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:14.729380 11016 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:14.805843  8576 rpcz_store.cc:276] 0709 13:02:12.919845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.951317 (+1031472us) service_pool.cc:224] Handling call
0709 13:02:13.951611 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.783749 (+832138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.784325 (+   576us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b5400440cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:14.784502 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1905783552}
I20260709 13:02:14.843205 10938 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:14.829159 11046 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:14.844200  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 479) took 2813 ms (client timeout 2948 ms). Trace:
I20260709 13:02:14.838223  8728 rpcz_store.cc:276] 0709 13:02:12.031763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.836918 (+2805155us) service_pool.cc:224] Handling call
0709 13:02:14.837187 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.837495 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.837516 (+    21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.837552 (+    36us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.837618 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.837665 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.837824 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:14.844708  8727 rpcz_store.cc:269] 0709 13:02:12.031086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.738284 (+2707198us) service_pool.cc:224] Handling call
0709 13:02:14.844127 (+105843us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.844786 11046 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.845885  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 486) took 2813 ms. Trace:
I20260709 13:02:14.845966 11046 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.846377 10938 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.846271  8728 rpcz_store.cc:276] 0709 13:02:12.032687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.845105 (+2812418us) service_pool.cc:224] Handling call
0709 13:02:14.845387 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.845577 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.845601 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.845650 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.845682 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.845733 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.845834 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.847041 10938 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.848331  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 490) took 2739 ms. Trace:
I20260709 13:02:14.718731 11035 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:14.827826  9330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 671) took 1885 ms (client timeout 2096 ms). Trace:
W20260709 13:02:14.750488  9326 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 666) took 2247 ms (client timeout 1929 ms). Trace:
I20260709 13:02:14.835155 11021 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:14.850126  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:14.850616 10938 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:14.702235  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.008s	sys 0.000s
I20260709 13:02:14.851179  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:14.842319  8650 rpcz_store.cc:276] 0709 13:02:13.373791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.049140 (+675349us) service_pool.cc:224] Handling call
0709 13:02:14.710147 (+661007us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.704835  8575 rpcz_store.cc:276] 0709 13:02:11.843058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.962440 (+1119382us) service_pool.cc:224] Handling call
0709 13:02:14.203798 (+1241358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.204062 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.204102 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.204162 (+    60us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:14.204212 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.204290 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.303428 (+ 99138us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400168ad0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.303620 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2853152640}
I20260709 13:02:14.841318  8583 rpcz_store.cc:276] 0709 13:02:11.770067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.770187 (+   120us) service_pool.cc:224] Handling call
0709 13:02:11.777649 (+  7462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.916145 (+138496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.313361 (+2397216us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54002cabd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:14.639955 (+326594us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":310194944}
I20260709 13:02:14.843132  8577 raft_consensus.cc:2410] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:14.841733  8655 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:14.760715  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.852850 10792 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:02:14.701982  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.004s	sys 0.004s
W20260709 13:02:14.851402  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.010s	sys 0.001s
I20260709 13:02:14.853515 10792 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:14.853374  9330 rpcz_store.cc:269] 0709 13:02:12.942940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.774105 (+1831165us) service_pool.cc:224] Handling call
0709 13:02:14.827762 (+ 53657us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.839556 10642 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:14.842873  8731 rpcz_store.cc:269] 0709 13:02:12.031301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.746617 (+2715316us) service_pool.cc:224] Handling call
0709 13:02:14.842100 (+ 95483us) spinlock_profiling.cc:243] Waited 48.7 ms on lock 0x7b540022a954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.842224 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111965952}
I20260709 13:02:14.851351 10938 raft_consensus.cc:2764] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:02:14.841988  8582 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:14.823148  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:14.850224  9326 rpcz_store.cc:269] 0709 13:02:12.502921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.503183 (+   262us) service_pool.cc:224] Handling call
0709 13:02:14.554905 (+2051722us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:14.750395 (+195490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6383872}
I20260709 13:02:14.842631  9329 rpcz_store.cc:276] 0709 13:02:12.797332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.797529 (+   197us) service_pool.cc:224] Handling call
0709 13:02:14.089739 (+1292210us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:14.703888 (+614149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3620608}
W20260709 13:02:14.853925  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 804630us
I20260709 13:02:14.821131  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.852946  8658 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:14.854082 10792 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:14.858170  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2222) took 1481 ms. Trace:
I20260709 13:02:14.858369  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:14.858541  8658 rpcz_store.cc:276] 0709 13:02:13.376840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.727253 (+1350413us) service_pool.cc:224] Handling call
0709 13:02:14.858109 (+130856us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.850435 11021 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.859562  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:14.859723 10990 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:14.860136 11021 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.729501 10911 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:14.854319  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.005s	sys 0.004s
W20260709 13:02:14.861510  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 900443us
I20260709 13:02:14.861759  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.853078  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:14.708997  8657 rpcz_store.cc:276] 0709 13:02:12.946821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.091736 (+144915us) service_pool.cc:224] Handling call
0709 13:02:13.092092 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.036784 (+944692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.037061 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.862591  8655 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:14.863008  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.864176  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.857477  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:14.860380  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.868772  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.869118  8581 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.869481  8659 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:14.852289  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.865113 11026 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:14.848754  8728 rpcz_store.cc:276] 0709 13:02:12.109224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.847434 (+2738210us) service_pool.cc:224] Handling call
0709 13:02:14.847710 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.848019 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.848046 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.848089 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.848130 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.848181 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.848279 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.784401 10617 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.860198  8658 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.880949  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 495) took 1873 ms. Trace:
I20260709 13:02:14.881165  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:14.881628  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 341) took 1934 ms. Trace:
I20260709 13:02:14.881342  9330 rpcz_store.cc:276] 0709 13:02:13.007127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.880072 (+1872945us) service_pool.cc:224] Handling call
0709 13:02:14.880348 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.880540 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.880570 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.880613 (+    43us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:14.880651 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.880702 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.880875 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.882099  8650 rpcz_store.cc:276] 0709 13:02:12.947099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.880892 (+1933793us) service_pool.cc:224] Handling call
0709 13:02:14.881144 (+   252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.881327 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.881351 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.881387 (+    36us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.881443 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.881489 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.881577 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.791730 10964 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:14.870476  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:14.892261  8659 raft_consensus.cc:2376] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 2. Re-sending same reply.
I20260709 13:02:14.856320  8654 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:14.857132  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:14.870764  8730 raft_consensus.cc:2368] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4.
I20260709 13:02:14.841264  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:14.871260  8727 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:14.894249  8651 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:14.895334  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 488) took 2862 ms. Trace:
W20260709 13:02:14.871757  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 45069.9us
I20260709 13:02:14.852885  8623 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:14.866806  8657 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:14.774382  8653 raft_consensus.cc:2376] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 4. Re-sending same reply.
I20260709 13:02:14.874934  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:14.809137 11054 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:14.825821  8622 connection.cc:256] RPC call timeout handler was delayed by -2.11693677400001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21960000000000002s for a response
W20260709 13:02:14.935894  8622 connection.cc:256] RPC call timeout handler was delayed by -2.1006158149999692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
W20260709 13:02:14.936299  8622 connection.cc:256] RPC call timeout handler was delayed by -1.5847146749999865s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260709 13:02:14.936676  8622 connection.cc:256] RPC call timeout handler was delayed by -1.5649146750000114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
W20260709 13:02:14.937141  8622 connection.cc:256] RPC call timeout handler was delayed by -1.542414674999975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16080000000000003s for a response
W20260709 13:02:14.937584  8622 connection.cc:256] RPC call timeout handler was delayed by -1.2544146749999641s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
W20260709 13:02:14.938030  8622 connection.cc:256] RPC call timeout handler was delayed by -1.22291467499997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
I20260709 13:02:14.773504  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:14.866796 10990 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 13:02:14.784125 11037 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.878301 11026 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:14.955488 11026 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:14.956365 10988 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:14.880301  9326 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:14.957639  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 496) took 1950 ms. Trace:
I20260709 13:02:14.958129  9326 rpcz_store.cc:276] 0709 13:02:13.007400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.879808 (+1872408us) service_pool.cc:224] Handling call
0709 13:02:14.880116 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.957307 (+ 77191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.957573 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.870167  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:14.737428  9323 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:02:14.887734  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:14.961081  8728 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:14.962424  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:14.963269  8728 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:14.749526  8729 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:14.964877  8729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 487) took 2932 ms (client timeout 3107 ms). Trace:
W20260709 13:02:14.965384  8729 rpcz_store.cc:269] 0709 13:02:12.032905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.748129 (+2715224us) service_pool.cc:224] Handling call
0709 13:02:14.964803 (+216674us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.967029  8729 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:14.967821  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1730) took 2625 ms. Trace:
I20260709 13:02:14.968276  8729 rpcz_store.cc:276] 0709 13:02:12.342423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.966245 (+2623822us) service_pool.cc:224] Handling call
0709 13:02:14.966619 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.967489 (+   870us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.967751 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:14.771705  9327 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 500) took 1763 ms (client timeout 1696 ms). Trace:
W20260709 13:02:14.969669  9327 rpcz_store.cc:269] 0709 13:02:13.008405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.072591 (+ 64186us) service_pool.cc:224] Handling call
0709 13:02:14.771620 (+1699029us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.898681  8656 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:14.896212  8727 rpcz_store.cc:276] 0709 13:02:12.033139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.845501 (+2812362us) service_pool.cc:224] Handling call
0709 13:02:14.845769 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.894354 (+ 48585us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.895129 (+   775us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.784025 11034 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.849473  8724 rpcz_store.cc:276] 0709 13:02:12.031515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.820327 (+2788812us) service_pool.cc:224] Handling call
0709 13:02:14.820607 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.820774 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.820798 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.820836 (+    38us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.820879 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.820929 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.821017 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.923342  8654 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:14.975585  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2206) took 1660 ms. Trace:
I20260709 13:02:14.976039  8654 rpcz_store.cc:276] 0709 13:02:13.315339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.922777 (+1607438us) service_pool.cc:224] Handling call
0709 13:02:14.923125 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.975297 (+ 52172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.975514 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.953828 11019 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:14.924688  8657 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:14.921860 10698 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:14.893872 11044 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:14.921864  8659 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:14.929852  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 492) took 2568 ms (client timeout 2481 ms). Trace:
I20260709 13:02:14.930333  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:14.892529 10964 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:14.929883 10938 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:14.978680  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2204) took 1668 ms. Trace:
W20260709 13:02:14.978497  8730 rpcz_store.cc:269] 0709 13:02:12.361241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.641260 (+2280019us) service_pool.cc:224] Handling call
0709 13:02:14.929743 (+288483us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.979043 10938 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:14.979408  8654 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:14.979580 10938 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:14.980072  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2211) took 1646 ms. Trace:
I20260709 13:02:14.980432  8654 rpcz_store.cc:276] 0709 13:02:13.334399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.977022 (+1642623us) service_pool.cc:224] Handling call
0709 13:02:14.977364 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.979833 (+  2469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.980018 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.887540 11020 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:14.933825 10980 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:14.934456 10888 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:02:14.951982 11054 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.004398 11019 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:14.722549 10798 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:15.007651 10798 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:14.875049 11035 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:14.879490 10811 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:15.010682  8724 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:14.882241 10839 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:14.953485  9325 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:02:14.775527  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 507) took 1689 ms (client timeout 1980 ms). Trace:
W20260709 13:02:14.814975 10879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.778s	user 0.009s	sys 0.017s
I20260709 13:02:14.973294  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1732) took 2630 ms. Trace:
I20260709 13:02:14.878875  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:15.030488  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 505) took 1963 ms. Trace:
I20260709 13:02:15.031023  9328 rpcz_store.cc:276] 0709 13:02:13.066966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.874212 (+1807246us) service_pool.cc:224] Handling call
0709 13:02:15.030403 (+156191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.029767  8727 rpcz_store.cc:276] 0709 13:02:12.342975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.972201 (+2629226us) service_pool.cc:224] Handling call
0709 13:02:14.972589 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.972827 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.972867 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.972924 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.972972 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.973046 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.973210 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.034016  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1737) took 2652 ms. Trace:
I20260709 13:02:15.034462  8727 rpcz_store.cc:276] 0709 13:02:12.381528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.032838 (+2651310us) service_pool.cc:224] Handling call
0709 13:02:15.033249 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.033490 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:15.033527 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.033598 (+    71us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:15.033643 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.033712 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.033943 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.983402  8729 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:02:15.037036  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1731) took 2694 ms. Trace:
I20260709 13:02:15.037493  8729 rpcz_store.cc:276] 0709 13:02:12.342705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.969207 (+2626502us) service_pool.cc:224] Handling call
0709 13:02:14.983059 (+ 13852us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.036708 (+ 53649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.036971 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.931248 11029 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:02:15.029799  9324 rpcz_store.cc:269] 0709 13:02:13.085853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.774386 (+1688533us) service_pool.cc:224] Handling call
0709 13:02:14.775460 (+  1074us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.052663 11061 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:15.065239  8731 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260709 13:02:15.065862  8731 pending_rounds.cc:85] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:02:15.066131  8731 pending_rounds.cc:107] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260709 13:02:15.066413  8731 raft_consensus.cc:2889] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:14.902047 10881 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.118s
I20260709 13:02:15.097899 10950 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:15.121654 10950 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:15.122817 10950 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.897298 10825 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.131s
I20260709 13:02:15.117859 10881 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:15.131346 10881 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:15.132675 10881 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.001519 11027 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:14.882591 11030 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.155s
I20260709 13:02:15.139000 11030 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:15.139578 11030 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:15.140865 11030 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:14.909554 10617 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.144578 10825 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:15.215685 10825 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:15.217052 10825 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.218879 10825 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:14.876392  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.278426  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:02:15.278901  8548 connection.cc:256] RPC call timeout handler was delayed by -1.4371280699999716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
I20260709 13:02:14.588884  8482 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 4, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 4 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:15.283494  8577 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:14.813320  8578 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:15.287006  8578 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:14.874876  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 79075.1us
I20260709 13:02:14.722378  8480 rpcz_store.cc:276] 0709 13:02:06.635195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.647679 (+ 12484us) service_pool.cc:224] Handling call
0709 13:02:14.603773 (+7956094us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:11.113289 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:11.197561 (+ 84272us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:11.197617 (+    56us) write_op.cc:435] Acquired schema lock
0709 13:02:11.197635 (+    18us) tablet.cc:662] Decoding operations
0709 13:02:11.246751 (+ 49116us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:11.253502 (+  6751us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:11.254639 (+  1137us) tablet.cc:685] Acquiring locks for 4 operations
0709 13:02:11.263874 (+  9235us) tablet.cc:701] Row locks acquired
0709 13:02:11.263893 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:02:11.264194 (+   301us) write_op.cc:270] Start()
0709 13:02:11.264327 (+   133us) write_op.cc:276] Timestamp: P: 1783602131264123 usec, L: 0
0709 13:02:11.264359 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 13:02:11.327018 (+ 62659us) log.cc:844] Serialized 1662 byte log entry
0709 13:02:11.694268 (+367250us) op_driver.cc:464] REPLICATION: finished
0709 13:02:11.758217 (+ 63949us) write_op.cc:301] APPLY: starting
0709 13:02:11.758334 (+   117us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:11.758542 (+   208us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:11.758731 (+   189us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:11.759782 (+  1051us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:11.760060 (+   278us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 13:02:11.760129 (+    69us) write_op.cc:312] APPLY: finished
0709 13:02:11.760397 (+   268us) log.cc:844] Serialized 41 byte log entry
0709 13:02:11.760584 (+   187us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:11.760692 (+   108us) write_op.cc:454] Released schema lock
0709 13:02:11.760734 (+    42us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:11.901739 (+141005us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b540012a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c11e28 00007f140d64c581 00007f140d64bd58 00007f140d6509f1 00007f140d6509a9 00007f140d650939 00007f140d650901 00007f140d64fb2d 000000000038f354 000000000038f199
0709 13:02:11.901956 (+   217us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:11.901997 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":146756,"reported_tablets":9,"spinlock_wait_cycles":666496,"child_traces":[["op",{"apply.queue_time_us":63647,"apply.run_cpu_time_us":2986,"apply.run_wall_time_us":3976,"num_ops":4,"prepare.queue_time_us":406295,"prepare.run_cpu_time_us":32685,"prepare.run_wall_time_us":796628,"raft.queue_time_us":117281,"raft.run_cpu_time_us":21670,"raft.run_wall_time_us":63841,"replication_time_us":429727,"spinlock_wait_cycles":135350528,"thread_start_us":406547,"threads_started":4,"wal-append.queue_time_us":245320,"wal-append.run_cpu_time_us":30189,"wal-append.run_wall_time_us":1188773}]]}
I20260709 13:02:14.950477  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:14.922377  8650 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:15.301103  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 419671us
I20260709 13:02:15.298326  8583 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:15.310300  8653 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:15.311259  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2215) took 1936 ms. Trace:
I20260709 13:02:15.311986  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2203) took 2001 ms. Trace:
I20260709 13:02:15.311733  8653 rpcz_store.cc:276] 0709 13:02:13.374453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.308985 (+1934532us) service_pool.cc:224] Handling call
0709 13:02:15.309810 (+   825us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.310915 (+  1105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.311152 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.983023  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2212) took 1637 ms. Trace:
I20260709 13:02:14.977917  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2208) took 1660 ms. Trace:
I20260709 13:02:14.981312  8659 rpcz_store.cc:276] 0709 13:02:13.313574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.921185 (+1607611us) service_pool.cc:224] Handling call
0709 13:02:14.921667 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.978429 (+ 56762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.978614 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.319818  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 369230us
I20260709 13:02:15.320155  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2063) took 3061 ms. Trace:
W20260709 13:02:15.325069  8699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.000s	sys 0.013s
I20260709 13:02:14.980158  8730 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 13:02:15.326431  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 456013us
I20260709 13:02:15.327368  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1733) took 2983 ms. Trace:
W20260709 13:02:14.805805  8725 rpcz_store.cc:269] 0709 13:02:12.029107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.239624 (+210517us) service_pool.cc:224] Handling call
0709 13:02:14.280843 (+2041219us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b54000893d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:14.281043 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48833792}
W20260709 13:02:15.326437  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 503881us
I20260709 13:02:15.353128 10964 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:15.354775 10950 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:02:15.347635  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 494473us
I20260709 13:02:15.313163  8650 rpcz_store.cc:276] 0709 13:02:13.310345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.899943 (+1589598us) service_pool.cc:224] Handling call
0709 13:02:14.922195 (+ 22252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.300402 (+378207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.311477 (+ 11075us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b540005a0d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:15.311765 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50244864}
W20260709 13:02:15.399745  8732 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.000s	sys 0.038s
I20260709 13:02:15.416049 11054 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:15.418807 10881 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:15.491266  8724 rpcz_store.cc:276] 0709 13:02:12.343266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.975386 (+2632120us) service_pool.cc:224] Handling call
0709 13:02:14.975782 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.011297 (+ 35515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.326603 (+315306us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b54001e73d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:15.327101 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79641472}
W20260709 13:02:15.434604  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:15.434772  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.009s	sys 0.030s
W20260709 13:02:15.441172  9331 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.013s	sys 0.023s
I20260709 13:02:15.441895 11049 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:02:15.442524  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 114194us
I20260709 13:02:15.444368 10617 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:15.446250 11034 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:15.447387  8578 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
W20260709 13:02:15.451834  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.453087  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:15.454917 11030 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:15.455554  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.458621  9323 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:15.459604 10916 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.706s	user 0.015s	sys 0.039s
I20260709 13:02:14.975051  8728 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:15.461537  8654 rpcz_store.cc:276] 0709 13:02:13.346147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.982006 (+1635859us) service_pool.cc:224] Handling call
0709 13:02:14.982371 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.982585 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.982637 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.982734 (+    97us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:14.982787 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.982861 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.982972 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.470337  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.407824  9322 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:15.482751 11027 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:15.497331  8699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.282s	user 0.000s	sys 0.014s
W20260709 13:02:15.500703 11053 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.750s	user 0.017s	sys 0.000s
I20260709 13:02:15.419145 11037 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:15.505337  8657 rpcz_store.cc:276] 0709 13:02:13.318383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.924104 (+1605721us) service_pool.cc:224] Handling call
0709 13:02:14.924465 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.977630 (+ 53165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.977855 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.490845  8731 rpcz_store.cc:276] 0709 13:02:12.258367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.855118 (+2596751us) service_pool.cc:224] Handling call
0709 13:02:14.855417 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.066962 (+211545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.319145 (+252183us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b5400417154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:15.319789 (+   644us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":481175296}
W20260709 13:02:15.510540 11043 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.765s	user 0.008s	sys 0.015s
W20260709 13:02:15.513993  9331 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.014s	sys 0.023s
I20260709 13:02:15.515126  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:15.518046  8732 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.000s	sys 0.039s
I20260709 13:02:15.520829  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1736) took 3153 ms. Trace:
W20260709 13:02:15.530738  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.009s	sys 0.030s
I20260709 13:02:15.527470  8578 pending_rounds.cc:85] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 2
W20260709 13:02:15.541162  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 238327us
W20260709 13:02:15.549631  8726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.027s	sys 0.014s
I20260709 13:02:15.549978 11053 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:15.551702  8656 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.580s	user 0.013s	sys 0.000s
I20260709 13:02:15.551518 11034 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.554791  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1738) took 3167 ms. Trace:
I20260709 13:02:15.557694  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2229) took 1609 ms. Trace:
W20260709 13:02:15.554884 11062 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.018s	sys 0.068s
W20260709 13:02:15.554729  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 64143.9us
I20260709 13:02:15.557976  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2223) took 2180 ms. Trace:
I20260709 13:02:15.581787  8650 rpcz_store.cc:276] 0709 13:02:13.377181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.504111 (+2126930us) service_pool.cc:224] Handling call
0709 13:02:15.557154 (+ 53043us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.557570 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:15.557610 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.557685 (+    75us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:15.557723 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.557781 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.557903 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.563601 10792 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.705s	user 0.005s	sys 0.013s
I20260709 13:02:15.506820 11049 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:15.508419 10811 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 13:02:15.581130  8654 rpcz_store.cc:276] 0709 13:02:13.948022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.543631 (+1595609us) service_pool.cc:224] Handling call
0709 13:02:15.556908 (+ 13277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.557276 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:15.557303 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.557370 (+    67us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:15.557430 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.557499 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.557615 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.591810  8726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.027s	sys 0.015s
I20260709 13:02:15.575698  8730 rpcz_store.cc:276] 0709 13:02:12.368306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.979516 (+2611210us) service_pool.cc:224] Handling call
0709 13:02:14.979824 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.519904 (+540080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.520543 (+   639us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.576450  8578 pending_rounds.cc:107] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 3.3
W20260709 13:02:15.535411  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 502) took 2515 ms (client timeout 1813 ms). Trace:
I20260709 13:02:15.593652  8578 raft_consensus.cc:2889] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:15.594671  8656 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
W20260709 13:02:15.591739 11062 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.019s	sys 0.069s
I20260709 13:02:15.594487 11049 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:15.607934 11034 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:15.614486  8733 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:15.614801 11034 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:15.618328  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 787853us
W20260709 13:02:15.594856  9323 rpcz_store.cc:269] 0709 13:02:13.019643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.301838 (+282195us) service_pool.cc:224] Handling call
0709 13:02:15.535179 (+2233341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.616778 11034 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:15.599479 10839 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:15.650924  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44542.4us
W20260709 13:02:15.609862  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 108714us
I20260709 13:02:15.621840  8659 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:15.545526 11029 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.115s
I20260709 13:02:15.626814  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:15.626572 11049 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:15.627010 11053 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:15.627521  8658 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.722s	user 0.011s	sys 0.004s
W20260709 13:02:15.627873  8580 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.728s	user 0.007s	sys 0.015s
I20260709 13:02:15.599805 10792 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:15.630638  9325 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.571s	user 0.017s	sys 0.005s
I20260709 13:02:15.656538  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:15.592860 11043 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.597164 11027 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.672369  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2224) took 2287 ms. Trace:
I20260709 13:02:15.654477 10642 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.580806  8727 rpcz_store.cc:276] 0709 13:02:12.386875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.035549 (+2648674us) service_pool.cc:224] Handling call
0709 13:02:15.035965 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:15.328945 (+292980us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:15.329014 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:15.330250 (+  1236us) log.cc:844] Serialized 26 byte log entry
0709 13:02:15.554022 (+223772us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b180038f980. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1
0709 13:02:15.554180 (+   158us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:15.554213 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.554347 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:15.554386 (+    39us) raft_consensus.cc:1685] finished
0709 13:02:15.554398 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.554625 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23956,"raft.queue_time_us":358170,"raft.run_cpu_time_us":508,"raft.run_wall_time_us":58618,"spinlock_wait_cycles":3298048,"thread_start_us":155873,"threads_started":1,"wal-append.queue_time_us":183652}
W20260709 13:02:15.671291  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 156407us
I20260709 13:02:15.678205  8659 rpcz_store.cc:276] 0709 13:02:13.399253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.538945 (+2139692us) service_pool.cc:224] Handling call
0709 13:02:15.611154 (+ 72209us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.662525 (+ 51371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.665321 (+  2796us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.667963 10917 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.781s	user 0.008s	sys 0.064s
I20260709 13:02:15.680837  8658 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:15.675884  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:15.648309 11054 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:15.679317  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:15.697672  8650 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:15.701574  8580 raft_consensus.cc:2410] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 13:02:15.707967  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45061.9us
I20260709 13:02:15.718073 11016 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:15.718917 11016 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:15.719367 11016 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:15.719772  9322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 508) took 2623 ms (client timeout 2988 ms). Trace:
I20260709 13:02:15.720356  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 485) took 3687 ms. Trace:
W20260709 13:02:15.720268  9322 rpcz_store.cc:269] 0709 13:02:13.096672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.875589 (+1778917us) service_pool.cc:224] Handling call
0709 13:02:15.718533 (+842944us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400552654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:15.719471 (+   938us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":847514880}
I20260709 13:02:15.721899  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 474) took 3692 ms. Trace:
I20260709 13:02:15.722193 10811 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:15.722839 10811 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:15.723872 10816 consensus_meta.cc:287] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.962s	user 0.016s	sys 0.011s
I20260709 13:02:15.724293 10811 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.720765  8733 rpcz_store.cc:276] 0709 13:02:12.032457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.842089 (+2809632us) service_pool.cc:224] Handling call
0709 13:02:14.842387 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.615123 (+772736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.720035 (+104912us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54001e64d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:15.720293 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1758251776}
I20260709 13:02:15.730608  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 343) took 1513 ms. Trace:
I20260709 13:02:15.722270  8726 rpcz_store.cc:276] 0709 13:02:12.029913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.767723 (+2737810us) service_pool.cc:224] Handling call
0709 13:02:14.767992 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.768186 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:14.948496 (+180310us) log.cc:844] Serialized 14 byte log entry
0709 13:02:15.686603 (+738107us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.708365 (+ 21762us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:15.721545 (+ 13180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.721671 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.721836 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42169,"prepare.queue_time_us":287344,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":54009,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":34788,"thread_start_us":813127,"threads_started":2,"wal-append.queue_time_us":589888}
W20260709 13:02:15.736479 10652 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.694s	user 0.013s	sys 0.013s
I20260709 13:02:15.737355 11035 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.737953  8729 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:15.738443  8729 pending_rounds.cc:85] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:15.738142 11035 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:15.740391  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1729) took 3278 ms. Trace:
I20260709 13:02:15.741662  8575 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:15.743528  8655 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.880s	user 0.006s	sys 0.020s
I20260709 13:02:15.743994  8655 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:15.744693 11054 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.738737  8729 pending_rounds.cc:107] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:15.702755 10642 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.710813  8653 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
W20260709 13:02:15.910841  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.740918  8728 rpcz_store.cc:276] 0709 13:02:12.342141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.964132 (+2621991us) service_pool.cc:224] Handling call
0709 13:02:14.974571 (+ 10439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.579502 (+604931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.620048 (+ 40546us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:16.013176  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:16.022456  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 413579us
W20260709 13:02:15.753471  8652 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.645s	user 0.011s	sys 0.006s
I20260709 13:02:15.762481  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1635) took 1006 ms. Trace:
W20260709 13:02:16.019553  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 392787us
I20260709 13:02:15.893791  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 484) took 3861 ms. Trace:
I20260709 13:02:15.707949  9325 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
W20260709 13:02:15.768249  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.021051  8729 raft_consensus.cc:2889] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:15.746831  8651 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:15.731076  8650 rpcz_store.cc:276] 0709 13:02:14.216810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.582760 (+1365950us) service_pool.cc:224] Handling call
0709 13:02:15.688351 (+105591us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.702333 (+ 13982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.730404 (+ 28071us) spinlock_profiling.cc:243] Waited 99.4 ms on lock 0x7b5400420c50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:15.730546 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228676352}
I20260709 13:02:15.967140  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:16.049657  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.046334  8580 rpcz_store.cc:276] 0709 13:02:14.724135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.782508 (+ 58373us) service_pool.cc:224] Handling call
0709 13:02:15.726256 (+943748us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.890333  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 280815us
W20260709 13:02:15.888376 10975 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.181s	user 0.004s	sys 0.039s
I20260709 13:02:15.745512  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:15.748492 10938 consensus_meta.cc:287] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.768s	user 0.014s	sys 0.029s
W20260709 13:02:15.752981 11020 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.684s	user 0.006s	sys 0.015s
W20260709 13:02:15.753762  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 80011.3us
W20260709 13:02:15.753373  8581 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.874s	user 0.016s	sys 0.009s
W20260709 13:02:15.965481 11017 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.187s	user 0.036s	sys 0.040s
W20260709 13:02:15.936090 11021 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.072s	user 0.011s	sys 0.012s
W20260709 13:02:16.062848  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 299403us
W20260709 13:02:15.877583 11046 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.790s	user 0.013s	sys 0.008s
I20260709 13:02:15.750826 10792 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:16.003671 10825 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.784s	user 0.014s	sys 0.007s
W20260709 13:02:15.768576 11026 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.812s	user 0.021s	sys 0.018s
I20260709 13:02:16.001268 11043 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:15.989599 11053 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.062172  8652 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:15.898423 10811 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:15.726431 10917 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:15.968582 10950 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:02:15.973802  9297 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.684s (SENT)
I20260709 13:02:15.724395 10816 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.106396  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 670) took 3171 ms. Trace:
W20260709 13:02:16.103418  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.746308 11034 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:16.047230  8732 rpcz_store.cc:276] 0709 13:02:12.032236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.841583 (+2809347us) service_pool.cc:224] Handling call
0709 13:02:14.843557 (+  1974us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.843766 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:15.625574 (+781808us) log.cc:844] Serialized 14 byte log entry
0709 13:02:15.836409 (+210835us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.893248 (+ 56839us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:15.893389 (+   141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.893512 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.893697 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":212097,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":7756,"raft.run_cpu_time_us":8512,"raft.run_wall_time_us":310928,"thread_start_us":770762,"threads_started":2,"wal-append.queue_time_us":100378}
W20260709 13:02:16.115788  8698 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.696s (SENT)
I20260709 13:02:16.267921  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2210) took 2945 ms. Trace:
W20260709 13:02:16.316254  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 214600us
I20260709 13:02:16.040625 10652 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.241434  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:15.757939 11057 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:16.115209 11049 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:16.123186  8726 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:16.007169 11027 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:16.322623 11085 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.020s	sys 0.042s
I20260709 13:02:16.072876  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:16.259116  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:16.317456  9331 rpcz_store.cc:276] 0709 13:02:12.935255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.882666 (+1947411us) service_pool.cc:224] Handling call
0709 13:02:14.882947 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.883145 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:15.595131 (+711986us) log.cc:844] Serialized 14 byte log entry
0709 13:02:16.062055 (+466924us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:16.081560 (+ 19505us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:16.105865 (+ 24305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.106089 (+   224us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:16.106295 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":517288,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":103,"raft.queue_time_us":16015,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":31601,"thread_start_us":995738,"threads_started":2,"wal-append.queue_time_us":675255}
I20260709 13:02:16.266429  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 494) took 1410 ms. Trace:
I20260709 13:02:16.099035  8584 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:16.102161  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:16.261896  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:16.098397  8581 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:16.098763 11068 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:16.327175  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 279162us
I20260709 13:02:16.240398 11088 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.103175  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:16.353286  8729 rpcz_store.cc:276] 0709 13:02:14.855999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.038440 (+182441us) service_pool.cc:224] Handling call
0709 13:02:15.038799 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.060969 (+1022170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.266080 (+205111us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x7b5400064854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:16.266349 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1605817600}
I20260709 13:02:16.334708 11054 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:16.335870  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 366327us
I20260709 13:02:16.098738 10938 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.098605  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:15.879213 10857 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:16.345528  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:16.319785  8651 rpcz_store.cc:276] 0709 13:02:13.322857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.930609 (+1607752us) service_pool.cc:224] Handling call
0709 13:02:14.930931 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.072244 (+1141313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.267571 (+195327us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b540016e4d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:16.267829 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1759486848}
I20260709 13:02:16.349969  8577 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
W20260709 13:02:16.319409  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 204380us
I20260709 13:02:16.306380  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2221) took 2929 ms. Trace:
I20260709 13:02:16.355473 11068 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.364127 10652 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.364451  8653 rpcz_store.cc:276] 0709 13:02:13.376483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.435225 (+2058742us) service_pool.cc:224] Handling call
0709 13:02:15.441389 (+  6164us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.048812 (+607423us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.265233 (+216421us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54000591d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:16.305422 (+ 40189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":583911680}
I20260709 13:02:16.365712 11049 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:16.365557 10857 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.366173 10950 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.366591 11049 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:16.366997 10950 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.368266 11049 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:16.359491 11053 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:16.359577  8548 connection.cc:256] RPC call timeout handler was delayed by -1.097767417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
I20260709 13:02:16.237474 11020 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.376374 11053 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.377442  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:16.369961 10938 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.083233 10975 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.370877 10816 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:16.099987 10642 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.371017 11030 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:16.110781  9325 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 501) took 3102 ms (client timeout 1962 ms). Trace:
I20260709 13:02:16.368268 10857 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:15.879431 10839 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:16.352587 11085 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.020s	sys 0.042s
I20260709 13:02:15.749728 11082 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.614s
I20260709 13:02:16.382431  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:16.394702  8733 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:16.400450  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:16.259490 11035 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.404520  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:16.398587  8576 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:16.356874 10825 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:16.098053 11016 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.412488  8726 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:16.413090  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.388882 11054 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.416328  8727 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:16.389637 10950 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:16.420876  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.351912 11026 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.405931  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:16.425486  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.425679  8729 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:16.427224  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.424508  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:16.359490 11088 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:16.433658  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:16.436728  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 469705us
W20260709 13:02:16.444804  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 78934.9us
W20260709 13:02:16.401548  9325 rpcz_store.cc:269] 0709 13:02:13.008630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.773224 (+1764594us) service_pool.cc:224] Handling call
0709 13:02:16.110558 (+1337334us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:16.448560  8730 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:16.449301  8731 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:16.450232  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:16.452195  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:16.454129 11054 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.455654 10975 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.456445 11054 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.459177 11026 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:16.459959  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.429371 10911 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:16.466518  8728 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:16.466765  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.469935  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 469) took 1060 ms. Trace:
I20260709 13:02:16.430510 10825 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.399286 11049 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260709 13:02:16.470441  8577 rpcz_store.cc:276] 0709 13:02:15.437088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.452335 (+ 15247us) service_pool.cc:224] Handling call
0709 13:02:16.469745 (+1017410us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54003449d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:16.469860 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1824036096}
W20260709 13:02:16.475860  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.439146 10816 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.417743 11030 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:16.481040  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 75244.5us
W20260709 13:02:16.489352  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 111022us
I20260709 13:02:16.463873 11053 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:16.493268  8581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1634) took 1765 ms (client timeout 1954 ms). Trace:
W20260709 13:02:16.493829  8581 rpcz_store.cc:269] 0709 13:02:14.723890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.751822 (+ 27932us) service_pool.cc:224] Handling call
0709 13:02:16.477125 (+1725303us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54002cbad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:16.488762 (+ 11637us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":267406720}
W20260709 13:02:16.494621  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 332664us
I20260709 13:02:16.486210 11017 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:16.485720 10857 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 13:02:16.498409 11017 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.499475 10857 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:16.499912 10857 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:16.499662 11030 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.501441 11017 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.501689 10839 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:16.475574 11089 raft_consensus.cc:2764] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:02:16.502653 11088 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:16.503501 10652 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.485741 10816 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.505954 11020 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.506546 10816 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.474828 10950 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:16.502354 10917 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:16.513167  8652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2214) took 3119 ms (client timeout 1962 ms). Trace:
I20260709 13:02:16.484169 11068 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:16.513912  8652 rpcz_store.cc:269] 0709 13:02:13.374172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.039756 (+665584us) service_pool.cc:224] Handling call
0709 13:02:16.476906 (+2437150us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b540025bf54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:16.493176 (+ 16270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2047964032}
I20260709 13:02:16.519652  8697 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:16.522107  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70060.2us
I20260709 13:02:16.496701 10825 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:16.473546 11086 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 13:02:16.530210  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:16.519048 10938 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.488241 10911 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.501199 10857 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.545528 11098 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:16.547375  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:16.547696 11016 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.548599 11016 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:16.552686 10825 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.553980 10825 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.554914 10811 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:16.556396 10825 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:16.535053 11068 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:16.577354 11057 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:16.564504  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.578207  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:16.581144 10950 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:16.583109  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:16.568490 11095 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:16.569576 10911 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:16.568961  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.589779 11095 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:16.590975  9325 raft_consensus.cc:2410] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:16.591271 11095 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.593475  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.539701 10938 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.570397  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.571199  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.596686  9322 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:16.596936 10938 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.598923  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.574280 11086 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:16.602005 10857 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:16.573154 11099 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260709 13:02:16.575929  8550 connection.cc:256] RPC call timeout handler was delayed by -3.3786320969999792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
W20260709 13:02:16.603843  8550 connection.cc:256] RPC call timeout handler was delayed by -3.3476002700000436s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
W20260709 13:02:16.604293  8550 connection.cc:256] RPC call timeout handler was delayed by -2.0549340009999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
W20260709 13:02:16.604588  8550 connection.cc:256] RPC call timeout handler was delayed by -2.032434000999956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2167s for a response
W20260709 13:02:16.604974  8550 connection.cc:256] RPC call timeout handler was delayed by -2.0243340009999997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
I20260709 13:02:16.581043  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:16.587374  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.587833  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:16.556733 10811 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:16.578339 11057 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.600325  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.600778  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:16.605316  8550 connection.cc:256] RPC call timeout handler was delayed by -1.0199340009999673s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3292s for a response
I20260709 13:02:16.566548  8724 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:16.614145  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.616674 11057 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.617602 11099 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:16.615338  9322 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:16.618068 11099 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:16.618564 11099 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:16.619398  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:16.620340 10938 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:16.621731  9331 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 2: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:16.622356 10938 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:16.622815 10938 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.622941 10911 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:16.623226 10938 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:16.623415 10911 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.623883 10911 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:16.608412 10811 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:16.626417  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:16.627485 11082 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:16.627923 11082 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.628474 11082 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:16.629987  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:16.630983 11030 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:16.631389 11030 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.631882 11030 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:16.626847 10811 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.633159 10811 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:16.638170  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55523.2us
I20260709 13:02:16.638837  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:16.638085 11082 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.639945  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:16.640841 10950 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:16.641458 10950 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:16.642064 11082 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:16.642941  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.643584  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:16.643719 11089 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:16.644192  9328 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:16.644395 11089 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:16.593300 11095 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:16.645846 11089 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.650426  9323 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:16.654822  9328 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:16.655913  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:16.655850 11030 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.642515 10950 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.657560  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.668692  9328 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:16.650650  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.680516  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:16.681437  8577 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:16.641480 10825 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:16.686408 10825 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.691515 10825 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:16.692649 11030 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:16.693444  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:16.694885 11082 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:16.699146  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:16.699527  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 160110us
I20260709 13:02:16.700146  8581 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:16.700740  8576 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:16.700721  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:16.643764  9322 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:16.702888  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:16.707417  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:16.711012  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.712922  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:16.667284 11098 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 13:02:16.719712  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:16.719949  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:16.720505  8583 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:16.718735  8698 leader_election.cc:336] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.779s (SENT)
I20260709 13:02:16.715240 11086 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:16.725473  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:16.707185 11082 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260709 13:02:16.728730 10642 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:16.729172 11034 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:16.729810 11034 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:16.729914 11086 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:16.730762 11034 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.731070 11086 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.732018 11034 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:16.731549 11107 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:16.729235 10642 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.736939 10642 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:16.726116  8582 raft_consensus.cc:2376] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3. Re-sending same reply.
I20260709 13:02:16.733965 11107 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:16.741035  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:16.747466  8548 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.679s (SENT)
I20260709 13:02:16.749171 11068 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.738883 10816 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:16.762019  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.762912  9325 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:16.763684 11016 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 13:02:16.764873 11016 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:16.765434 11016 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:16.759644 11098 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:16.763881  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:16.769745  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50404.9us
W20260709 13:02:16.760659  8698 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.813s (SENT)
I20260709 13:02:16.801467  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:16.809641  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62201.9us
I20260709 13:02:16.758939 10816 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.804641  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.846858  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:16.847273  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.847323  9328 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
W20260709 13:02:16.850872  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 91633.3us
W20260709 13:02:16.853123  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.855228  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:16.781687 11098 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:16.802836 10792 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:16.812760 10652 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:16.860813  8548 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.802s (SENT)
I20260709 13:02:16.858167  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:16.869621  8727 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:16.766844 11095 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
W20260709 13:02:16.872238  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 159329us
I20260709 13:02:16.872735  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.841425 10816 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:16.873895  9328 raft_consensus.cc:1813] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Handling vote request from an unknown peer a0d703ee18464475a290425976246555
I20260709 13:02:16.874735  9328 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 2: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:16.877961  8698 leader_election.cc:336] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.825s (SENT)
I20260709 13:02:16.876322  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.879788  8652 raft_consensus.cc:2368] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3.
I20260709 13:02:16.823838  9322 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:16.884016  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.804836 11068 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:16.878587  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 96306.8us
I20260709 13:02:16.803656  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.864289 11046 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:16.865440  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.812489 11020 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260709 13:02:16.766757 11016 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.892058  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.910619  8650 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:16.907377  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:16.909413  8658 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:16.911595  8727 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:16.909559  9331 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
W20260709 13:02:16.913604  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.882390  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.890594 11057 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.864712 11099 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.899217 11053 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:16.875377 11107 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:16.879398 10652 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.934324  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:16.914182 11113 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.935810  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.900098  9323 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:16.890475 11098 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.937747 11110 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:16.937958  9328 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:02:16.937994  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.939006  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.939877  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:16.941196  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.931413 11057 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:16.950506  8698 leader_election.cc:336] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.870s (SENT)
I20260709 13:02:16.941852  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:16.956866  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:17.000218  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.913556 11094 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 13:02:16.920924  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:16.966724  9322 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:16.928602  9325 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:16.975317  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 126166us
I20260709 13:02:16.977329  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.990506  8656 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:16.932651 11035 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:16.941252 10825 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.988106  9329 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:16.994021  8650 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:17.007295  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:16.997522 11114 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 13:02:16.923516 11027 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:02:17.026763  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.027264  9296 leader_election.cc:400] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3.
I20260709 13:02:16.993896 11110 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:17.041684  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:17.045135  9296 leader_election.cc:403] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:17.047170  8655 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:17.053648  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:17.059991  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:17.058138  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:17.060915  8650 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:17.065097 11109 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.182s	user 0.000s	sys 0.002s
I20260709 13:02:17.061121  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:17.070251  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 138669us
I20260709 13:02:17.020205  8653 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
W20260709 13:02:17.072428  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 126858us
I20260709 13:02:17.058905 10798 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.007189 11026 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 13:02:17.063479  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.063792  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.063484  9328 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:17.060015  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:17.062083 11068 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:17.032168 11016 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:17.026239 11035 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:17.086767  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45128.9us
I20260709 13:02:17.028677 10792 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:17.032580  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:17.085294  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:17.096863  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 140201us
I20260709 13:02:17.015122 10911 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.087953 11030 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:17.097795  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.115206  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:17.098023  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.098696  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:17.048889 11049 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:17.123339  9329 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:17.099231 10938 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:16.992961 11053 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.081295  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:17.098723  8654 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:17.107403  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:16.996639 11113 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.114153  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:17.130540  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:17.130590  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.132269  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.135128  8659 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 2: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:17.048601 11089 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260709 13:02:17.143128  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:17.099507 11111 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
W20260709 13:02:17.174000  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51296.5us
I20260709 13:02:17.127949 11049 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:17.149076 10816 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.133596  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:17.153367  8657 raft_consensus.cc:2376] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4. Re-sending same reply.
I20260709 13:02:17.102236 10798 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:17.165206  9323 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:17.187119  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:17.135138  8653 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:17.181618 11123 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:17.207759  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.128955 10642 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:17.166008 11099 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:17.211709  8654 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:17.211994  9328 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:17.212878  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.186412 11049 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:17.225054  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 143850us
I20260709 13:02:17.203728  9324 raft_consensus.cc:2368] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5.
I20260709 13:02:17.189757 11088 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:02:17.227967  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:17.228467  8697 connection.cc:256] RPC call timeout handler was delayed by -1.7789346280000018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2988s for a response
I20260709 13:02:17.228657  8650 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:17.231209  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:17.235031  8581 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:17.235641  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.240023 10792 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:17.243595 10938 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:17.244356 10938 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:17.245750 10798 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:17.240091 11123 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:17.247509 11053 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:17.248309 10825 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:17.248217 11123 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.249764  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:17.250361 10816 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:17.250586  8577 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:17.275688  8583 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.549s	user 0.012s	sys 0.003s
I20260709 13:02:17.316123  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:17.319029  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 204925us
I20260709 13:02:17.317158  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:17.244915 10938 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:17.320844  8582 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
W20260709 13:02:17.320950 11122 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 13:02:17.238852 11111 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:17.322301 11111 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.322901 11111 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:17.323472  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.324932  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.325685 10816 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:17.326113 10816 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:17.326385 11113 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:17.327036 10816 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.267118  9325 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:17.328373  8654 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:17.328862  8650 raft_consensus.cc:2368] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3.
I20260709 13:02:17.346143  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:17.355170  8581 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:17.360069  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:17.240245 11088 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:17.316502  8583 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:17.330295  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:17.330597  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 101364us
I20260709 13:02:17.366196  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:17.363281  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:17.618727  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:17.266305 10652 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:17.331954  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.331804 11053 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 13:02:17.360244 11099 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 13:02:17.393172  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:17.393437  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:17.213343 11110 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.393887  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.210026 11035 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:17.396446  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 169970us
W20260709 13:02:17.397147  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.371255  8582 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:17.419713  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 318065us
I20260709 13:02:17.431329 10798 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:17.431701 11118 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:17.432468 10911 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:17.438225 11088 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:17.459652  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 60121.2us
I20260709 13:02:17.364266 11057 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:17.437121 11030 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3.
I20260709 13:02:17.792837  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:17.507257  8657 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:17.527668 11126 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 13:02:17.528581  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:17.567233  8652 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:17.527386 11137 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.794993  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:17.818308 11137 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.467227 11129 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:17.617004  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 185624us
I20260709 13:02:17.621742  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:17.821503 11057 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:17.822183 11057 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:17.823206 11057 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:17.825057 11111 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.006s
I20260709 13:02:17.768930  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:17.797329  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.828681 11111 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:17.621644  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:17.831642 11111 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:17.833674 11133 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:17.831656  8584 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:17.834558 11133 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.835282 11133 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:17.662879  8727 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.740s	user 0.012s	sys 0.000s
I20260709 13:02:17.804011  8724 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:17.818696  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:17.620498  8731 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:17.785532  9322 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.743s	user 0.011s	sys 0.002s
I20260709 13:02:17.825565  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:17.619379  8729 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:18.017284  8584 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:17.898358  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.155571  8727 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:17.617009 10816 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:18.156430  8549 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.096s (SENT)
W20260709 13:02:17.898617  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:17.945322  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.839985  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:18.165939  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.062664  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.854s	user 0.019s	sys 0.016s
W20260709 13:02:18.166702  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.022s	sys 0.018s
I20260709 13:02:18.158494  8582 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 13:02:18.030933 11049 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.783s	user 0.010s	sys 0.003s
I20260709 13:02:17.863731  8650 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:17.897881  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 504418us
I20260709 13:02:17.994168  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:17.998291  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:18.155294  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 334134us
I20260709 13:02:17.912755  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:17.771253 11016 consensus_meta.cc:287] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.664s	user 0.007s	sys 0.003s
I20260709 13:02:18.165625  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1763) took 1269 ms. Trace:
I20260709 13:02:17.685529  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.885340  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.730917 11068 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:18.166533  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 326574us
I20260709 13:02:18.169291 11030 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:18.018265 11123 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.003s	sys 0.030s
I20260709 13:02:17.666432 10642 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:17.781054 10825 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:18.166517  8696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.737s (SENT) [suppressed 51 similar messages]
I20260709 13:02:18.180073  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.171957  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:18.187420  8727 rpcz_store.cc:276] 0709 13:02:16.887345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.893804 (+  6459us) service_pool.cc:224] Handling call
0709 13:02:18.156238 (+1262434us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.163511 11137 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:18.164458  9322 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:18.078152 11111 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.190493  8728 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:18.189116  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.202950  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.050559 11134 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:18.166170 11107 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.000s	sys 0.017s
I20260709 13:02:18.165431 11129 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:18.193917  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2237) took 1042 ms. Trace:
I20260709 13:02:18.246850  8657 rpcz_store.cc:276] 0709 13:02:17.151806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.884915 (+733109us) service_pool.cc:224] Handling call
0709 13:02:18.193866 (+308951us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.203125  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:17.844969 11099 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:18.187644 11035 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:18.190610 11123 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.009s	sys 0.031s
I20260709 13:02:18.199154  8726 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:18.248555  8653 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:18.249239 11057 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:18.249504  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2238) took 1097 ms. Trace:
I20260709 13:02:18.249563  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.249958 11057 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.199066  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.248814  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:18.200935  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.188709  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.201813 10825 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.207710  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.209833  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:18.220902  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 697) took 1279 ms. Trace:
W20260709 13:02:18.223021  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 692) took 1624 ms (client timeout 1965 ms). Trace:
I20260709 13:02:18.223510 10798 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:18.224416  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.259414  8728 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:18.225260  8698 leader_election.cc:400] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3.
I20260709 13:02:18.229476  8584 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
W20260709 13:02:18.230322  9297 connection.cc:256] RPC call timeout handler was delayed by -1.2229465429999777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260709 13:02:18.232288  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.262759  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:18.188534 11068 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:18.194370  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.191633  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.250182  8653 rpcz_store.cc:276] 0709 13:02:17.152056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.202850 (+1050794us) service_pool.cc:224] Handling call
0709 13:02:18.249439 (+ 46589us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:17.842135 11131 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.392s
I20260709 13:02:18.199890  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:18.252064  8650 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:18.268450  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2244) took 1042 ms. Trace:
I20260709 13:02:18.268937  8650 rpcz_store.cc:276] 0709 13:02:17.226098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.195774 (+969676us) service_pool.cc:224] Handling call
0709 13:02:18.268374 (+ 72600us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.272338  9328 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:18.253084  8727 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:18.189589 11049 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.237815 10938 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.256353  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:18.245429 11107 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.000s	sys 0.022s
I20260709 13:02:18.257231  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2245) took 1022 ms. Trace:
I20260709 13:02:18.274891  8658 raft_consensus.cc:2368] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:18.274760  8652 rpcz_store.cc:276] 0709 13:02:17.235127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.207424 (+972297us) service_pool.cc:224] Handling call
0709 13:02:18.257177 (+ 49753us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.202639 11140 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:18.259898  8698 leader_election.cc:403] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:02:18.204980 11137 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 13:02:18.277173  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.277846  8653 raft_consensus.cc:1813] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Handling vote request from an unknown peer a0d703ee18464475a290425976246555
I20260709 13:02:18.278357  8653 raft_consensus.cc:2368] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3.
I20260709 13:02:18.278748  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2128) took 1186 ms. Trace:
I20260709 13:02:18.279052  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:18.280962  8650 raft_consensus.cc:2368] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:18.281260  8582 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:18.237619 11016 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 13:02:18.239382 11111 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.263170  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.279464 11140 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:18.283861  9330 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:18.283670  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:18.284358  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 84550.8us
I20260709 13:02:18.285348  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.286751  8657 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:18.282125  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.254066  9322 rpcz_store.cc:276] 0709 13:02:16.927461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.933964 (+  6503us) service_pool.cc:224] Handling call
0709 13:02:18.206711 (+1272747us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.314998  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.316840  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.325090  8724 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:18.323578  8650 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:18.288820  8731 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:18.326555  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1768) took 1010 ms. Trace:
I20260709 13:02:18.266467  8725 raft_consensus.cc:1813] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Handling vote request from an unknown peer a0d703ee18464475a290425976246555
I20260709 13:02:18.327035  8731 rpcz_store.cc:276] 0709 13:02:17.329171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.345811 (+ 16640us) service_pool.cc:224] Handling call
0709 13:02:18.326465 (+980654us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.327885  8726 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:18.289434  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.325117  8658 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:18.331205  8652 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:18.329768  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2127) took 1248 ms. Trace:
I20260709 13:02:18.331008  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.331915  8658 rpcz_store.cc:276] 0709 13:02:17.081650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.283087 (+1201437us) service_pool.cc:224] Handling call
0709 13:02:18.329676 (+ 46589us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.332137  8727 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:18.333163  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.333686  8727 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:18.334272  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.257647 11134 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.277187  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:18.339143  8584 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:18.341568  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.322878 10938 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:02:18.256523  9324 rpcz_store.cc:269] 0709 13:02:16.598902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.599963 (+  1061us) service_pool.cc:224] Handling call
0709 13:02:18.222747 (+1622784us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b5400551c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:18.222932 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1211893504}
I20260709 13:02:18.345376  8727 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:18.347450 11134 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.349566  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.349974 10825 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.351107 11057 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:18.352085 11049 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.358186  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.376077  8583 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:18.360383  9297 leader_election.cc:336] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.533s (SENT)
I20260709 13:02:18.371066  8730 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:18.330090 11016 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:18.372257  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.279091  8653 rpcz_store.cc:276] 0709 13:02:17.092467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.276858 (+1184391us) service_pool.cc:224] Handling call
0709 13:02:18.278691 (+  1833us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.379499  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.383459  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.383845  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:18.384285  8724 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
W20260709 13:02:18.384207  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 217637us
I20260709 13:02:18.384883  8724 pending_rounds.cc:85] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 2
I20260709 13:02:18.378602  9323 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:02:18.385912  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 160737us
I20260709 13:02:18.368289 10950 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.385926  8724 pending_rounds.cc:107] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:02:18.398890  8724 raft_consensus.cc:2889] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:18.376448  8725 raft_consensus.cc:2368] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3.
I20260709 13:02:18.333792 11133 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.400254  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.401298  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.402098  8584 raft_consensus.cc:2368] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:18.403541  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:18.376039 11134 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:18.375288 10938 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:18.380616 11016 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:18.415498  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:18.427469  8651 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:18.382295 11057 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.428078  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.386767 11099 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.428660  8582 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:18.432659  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 350) took 1573 ms (client timeout 1752 ms). Trace:
W20260709 13:02:18.433133  8651 rpcz_store.cc:269] 0709 13:02:16.854501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.863714 (+  9213us) service_pool.cc:224] Handling call
0709 13:02:18.415344 (+1551630us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:18.428094 (+ 12750us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3094272}
W20260709 13:02:18.433197  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 49417.1us
I20260709 13:02:18.433135  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.434718  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:18.436185  8583 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:18.436892  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2223) took 1038 ms. Trace:
I20260709 13:02:18.340370 11068 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.415910 11123 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:18.408366 11148 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.438766  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.439146  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:18.440299  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58349.6us
I20260709 13:02:18.438851  8579 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:18.448722  9297 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.552s (SENT)
I20260709 13:02:18.443912  8727 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:18.408859 10798 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3.
I20260709 13:02:18.437642  8583 rpcz_store.cc:276] 0709 13:02:17.398836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.430960 (+ 32124us) service_pool.cc:224] Handling call
0709 13:02:18.357982 (+927022us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:18.436833 (+ 78851us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4751232}
I20260709 13:02:18.456462  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:18.457967  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.458580  8582 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:18.430977 11147 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:18.463229  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.437647 11035 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:18.455085  9297 leader_election.cc:336] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.083s (SENT)
I20260709 13:02:18.466521  9297 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:18.466970  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:18.464987  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.470360 11054 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:18.471464 11054 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.439132 11140 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.475133  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.427500 11134 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.482903 11053 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:18.468927 11153 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:18.483507 11053 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:18.483992  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:18.483858 11153 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.484990  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:18.478838  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.477723 11082 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:18.438730 11099 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.481108 11016 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.425876 10950 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.482402 11111 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.478471  8582 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:18.486528  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.487703  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.459432 11049 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:18.468479 11131 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:18.466629 11068 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:18.495513  8583 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:18.497072  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:18.503825  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.438540 11057 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.438416 10938 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.479768  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:18.508277  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 53235.9us
I20260709 13:02:18.490473  8727 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:18.506126  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:18.508934  9297 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.481s (SENT)
W20260709 13:02:18.499361  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52065.1us
I20260709 13:02:18.501493  8728 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:18.509147  8579 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:18.509977 11160 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:18.510265  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:18.510684  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50036.5us
I20260709 13:02:18.511435  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:18.511567  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.512252  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:18.510339  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.509377  9297 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:18.505839 11054 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:18.514129 11110 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:18.514171 11054 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:18.514755 11110 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:18.514796 11054 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:18.515265 11110 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:18.515322 11054 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:18.516973  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.510428 11160 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:18.466153 11148 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:18.523387  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 72594.7us
I20260709 13:02:18.505287 11035 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:18.524564  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.525580  8582 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:18.484803 11053 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.527118  8579 raft_consensus.cc:2376] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5. Re-sending same reply.
I20260709 13:02:18.527581 11053 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:18.527710  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.528230  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.528674  8582 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:18.528920  8579 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:18.507864 11111 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:18.522477 10816 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:18.530879  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.530869  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.523125 11160 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:18.532167 11099 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:18.532846  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.533492 11153 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:18.527154 11035 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:18.546890  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.539057  8725 raft_consensus.cc:2368] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3.
I20260709 13:02:18.538268 11111 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:18.469559 11147 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:18.642238  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 77078.5us
W20260709 13:02:18.724146  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.676311  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:18.568534  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 83605.9us
W20260709 13:02:18.740836  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 227538us
I20260709 13:02:18.748062  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:18.748267  8724 raft_consensus.cc:2368] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4.
I20260709 13:02:18.521714 11123 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:18.745555  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 234159us
I20260709 13:02:18.746157  8730 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:02:18.590350  8579 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:18.576403  8583 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:18.752786  8578 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:18.645671  8729 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
W20260709 13:02:18.756285  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.754570  8550 leader_election.cc:336] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.645s (SENT)
I20260709 13:02:18.756953  8623 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:18.756790  8698 connection.cc:256] RPC call timeout handler was delayed by -1.85145576299999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23550000000000001s for a response
I20260709 13:02:18.772399  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:18.762301  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.772965  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.774551  8728 raft_consensus.cc:2410] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 13:02:18.774781  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.773017  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.688406 11053 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.783689  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:18.757265  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:18.783967  8698 connection.cc:256] RPC call timeout handler was delayed by -1.8449416740000402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
I20260709 13:02:18.784860  8583 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:18.785213  8698 connection.cc:256] RPC call timeout handler was delayed by -1.7389557629999786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.248s for a response
W20260709 13:02:18.786908  8698 connection.cc:256] RPC call timeout handler was delayed by -1.657594692000032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2344s for a response
W20260709 13:02:18.787283  8698 connection.cc:256] RPC call timeout handler was delayed by -1.5007276779999756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260709 13:02:18.787614  8698 connection.cc:256] RPC call timeout handler was delayed by -1.4820946920000324s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2539s for a response
W20260709 13:02:18.787946  8698 connection.cc:256] RPC call timeout handler was delayed by -1.2649276779999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
I20260709 13:02:18.558895 11026 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:18.737684 11098 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:18.554747 11148 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.794291  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:18.755802 11111 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:18.755599 11159 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 13:02:18.556054 11134 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:18.803398  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.752445 11016 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:18.798728  8582 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:18.661079 11156 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.754750 11133 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:18.754958 10950 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.782943 11158 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:18.792966 11123 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:18.806111  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.730872 11054 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:18.810987  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 302003us
W20260709 13:02:18.811501  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56954.1us
I20260709 13:02:18.793480 11053 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.818989  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:18.792222 11026 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:18.832351  8550 leader_election.cc:336] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.824s (SENT)
I20260709 13:02:18.832975  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:18.833904  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.799592 11153 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:18.881040  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 124124us
W20260709 13:02:18.882225  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 49321.9us
W20260709 13:02:18.886807  9297 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.948s (SENT)
I20260709 13:02:18.887603  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:18.888268  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 79666.7us
I20260709 13:02:18.888617  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:18.890154  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:18.890733  8726 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:18.891777  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:18.835180 11035 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:18.893079 11110 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.894121 11140 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.894754 11140 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:18.896018 11110 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:18.835925 11134 raft_consensus.cc:2764] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:02:18.897737  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.898679  8583 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:18.899103  8623 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:18.900882  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:18.809665 11162 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:18.899663  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:18.919148  8550 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:18.919411  8583 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:18.911960  8578 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:18.920320  8582 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
W20260709 13:02:18.920789  8578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2226) took 1482 ms (client timeout 1963 ms). Trace:
W20260709 13:02:18.873713  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.917924  8728 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:18.922098  8549 leader_election.cc:400] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3.
I20260709 13:02:18.922894  8549 leader_election.cc:403] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:02:18.921502  8578 rpcz_store.cc:269] 0709 13:02:17.438456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.440638 (+  2182us) service_pool.cc:224] Handling call
0709 13:02:18.546718 (+1106080us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:18.920707 (+373989us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2377472}
W20260709 13:02:18.955263  8698 leader_election.cc:336] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.034s (SENT)
I20260709 13:02:18.897224 11140 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:18.956532  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66434.1us
I20260709 13:02:18.816466 11158 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:18.966761  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.964880  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:18.961192  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 60417.2us
I20260709 13:02:18.836277 11099 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.966099  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.921590 11162 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:18.958686  8729 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:18.967118  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46126.7us
I20260709 13:02:18.891793 11147 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:18.901170 11049 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:02:18.817965 11156 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:19.032099  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 76863.2us
W20260709 13:02:19.036561  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 204178us
I20260709 13:02:19.027733  8579 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:19.051214  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:18.952440 10816 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:19.049223  8582 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:19.055611  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2237) took 1235 ms. Trace:
I20260709 13:02:19.054344  8583 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:18.925689 10811 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.902559 10938 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:18.905121 11148 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:19.056143  8582 rpcz_store.cc:276] 0709 13:02:17.819961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.966338 (+1146377us) service_pool.cc:224] Handling call
0709 13:02:19.055519 (+ 89181us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:19.065670  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 234036us
I20260709 13:02:19.053835 11153 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:18.957023 11160 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.012619 11134 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 1 FOLLOWER]: Advancing to term 3
W20260709 13:02:19.068626  8698 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.344s (SENT)
I20260709 13:02:19.028638 11035 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:19.039769 11068 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:19.004832 11133 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:19.072490  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.070011  8583 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:18.973665 11158 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.054805 11162 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.082477  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2234) took 1263 ms. Trace:
I20260709 13:02:19.079596  8582 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:19.091158  8583 rpcz_store.cc:276] 0709 13:02:17.819035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.066820 (+1247785us) service_pool.cc:224] Handling call
0709 13:02:19.069696 (+  2876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.082170 (+ 12474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.082413 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:19.092896  8582 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:02:19.093932  8727 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.068410 11148 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:19.068847 10811 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.085232 11166 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:19.075704 11049 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:19.079190 11035 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:19.069389 11099 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.071848 10839 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:19.078796 10938 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:19.106338 11148 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:19.114794  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46164us
I20260709 13:02:19.068298 11171 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:19.105914 11133 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.128998  8727 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:19.064674 11026 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.109328 11173 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:19.110865 11166 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:19.071694 11153 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.116118 11035 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:19.145749  8698 leader_election.cc:336] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.539s (SENT)
W20260709 13:02:19.148660  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 94371.4us
W20260709 13:02:19.151594  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.236127  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 166330us
W20260709 13:02:19.237178  9297 leader_election.cc:336] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 3.093s (SENT)
I20260709 13:02:19.237739  9297 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:19.133378 11173 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:19.119831 11049 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.139274 11166 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.130040 11054 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:19.261315  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 111510us
W20260709 13:02:19.265754  8550 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.142s (SENT)
I20260709 13:02:19.132547 10839 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.310011  8623 leader_election.cc:400] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5.
I20260709 13:02:19.348438  8623 leader_election.cc:403] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:19.134053 11133 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.358335 11171 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:19.245682 10816 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:19.264235 11026 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:19.366652  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 129495us
W20260709 13:02:19.368686  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 102969us
W20260709 13:02:19.369510  8550 leader_election.cc:336] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.167s (SENT)
W20260709 13:02:19.370321  8550 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.176s (SENT)
I20260709 13:02:19.370672  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:19.376127 10811 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:19.378001 11160 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.378942 10938 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:19.393561  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 428634us
W20260709 13:02:19.412572  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 275218us
W20260709 13:02:19.413972  8698 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.691s (SENT)
W20260709 13:02:19.414018  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43711.1us
W20260709 13:02:19.414883  8698 leader_election.cc:336] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.988s (SENT)
W20260709 13:02:19.412097  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 102308us
W20260709 13:02:19.412724 11111 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.001s
I20260709 13:02:19.395681 11082 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260709 13:02:19.418160  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:19.412858 10816 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.408910 11156 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:19.427165  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.435039 11082 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:19.438694  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.417073 11111 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.449895 11082 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:19.428354 11169 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
W20260709 13:02:19.450819 11165 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.377s	user 0.001s	sys 0.000s
I20260709 13:02:19.428545 10816 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.426388 11187 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:02:19.462316  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:19.466570  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.372457 11173 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.470567  8583 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:19.458431 11123 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:02:19.463330 11016 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.654s	user 0.010s	sys 0.001s
I20260709 13:02:19.477001 11173 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:19.477695 10652 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:19.480178 11153 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.458600 11171 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.481580 11123 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:19.495903 10938 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.496789 11123 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.471426 10816 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:19.498368  8576 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:19.499225  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2225) took 2061 ms. Trace:
I20260709 13:02:19.499585  8576 rpcz_store.cc:276] 0709 13:02:17.438018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.476934 (+ 38916us) service_pool.cc:224] Handling call
0709 13:02:19.497983 (+2021049us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:19.498183 (+   200us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.498881 (+   698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.499130 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394551040}
I20260709 13:02:19.481734 11150 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.500842  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.467760 11156 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:19.501092 11150 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.504865 11150 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:19.484277  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 512) took 1002 ms. Trace:
I20260709 13:02:19.484524  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.506381  8580 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:19.505575  8730 rpcz_store.cc:276] 0709 13:02:18.481747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.755971 (+274224us) service_pool.cc:224] Handling call
0709 13:02:18.772493 (+ 16522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.772764 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.772789 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:18.781979 (+  9190us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.920840 (+138861us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:18.920876 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.920946 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.484060 (+563114us) raft_consensus.cc:1685] finished
0709 13:02:19.484079 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.484217 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38016,"thread_start_us":138422,"threads_started":1,"wal-append.queue_time_us":183932}
W20260709 13:02:19.506953  8580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2228) took 2067 ms (client timeout 2064 ms). Trace:
W20260709 13:02:19.507365  8580 rpcz_store.cc:269] 0709 13:02:17.439196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.440772 (+  1576us) service_pool.cc:224] Handling call
0709 13:02:19.484375 (+2043603us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:19.506881 (+ 22506us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4880640}
W20260709 13:02:19.485638 11098 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.685s	user 0.008s	sys 0.004s
I20260709 13:02:19.496722 10938 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.462644 11099 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:19.486690 11118 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:19.511792 11118 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.513007 10839 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:19.492758  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:19.522022  9327 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 1.
W20260709 13:02:19.522397  8697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.066s	sys 0.064s
W20260709 13:02:19.522686  9327 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 517) took 2585 ms (client timeout 1772 ms). Trace:
I20260709 13:02:19.497462 11175 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:19.502362  8576 raft_consensus.cc:2376] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4. Re-sending same reply.
I20260709 13:02:19.523581  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.524417  8577 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:19.524300 11175 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.482646 11133 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.526278  8581 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:19.527000  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2221) took 2165 ms. Trace:
I20260709 13:02:19.527383  8581 rpcz_store.cc:276] 0709 13:02:17.361578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.361739 (+   161us) service_pool.cc:224] Handling call
0709 13:02:19.525921 (+2164182us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:19.526099 (+   178us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.526696 (+   597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.526910 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2612608}
W20260709 13:02:19.483549 11170 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.410s	user 0.000s	sys 0.002s
I20260709 13:02:19.528838  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:19.531271  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 511) took 1068 ms. Trace:
W20260709 13:02:19.531862  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.532327  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.533605  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.33758e+06us
I20260709 13:02:19.536161 11162 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.538422  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.538692 11098 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.541265 11134 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3.
I20260709 13:02:19.527484 11187 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5.
I20260709 13:02:19.542725  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:19.546639  9331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 522) took 2466 ms (client timeout 1518 ms). Trace:
W20260709 13:02:19.547272  9331 rpcz_store.cc:269] 0709 13:02:17.080287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.106273 (+ 25986us) service_pool.cc:224] Handling call
0709 13:02:19.528594 (+2422321us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:19.546527 (+ 17933us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9521024}
I20260709 13:02:19.533576 11175 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.510885  8654 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:19.510998 11111 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:19.514112  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.554742  8580 raft_consensus.cc:2376] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4. Re-sending same reply.
I20260709 13:02:19.485894 11049 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.514257 11118 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:19.514971  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 604327us
I20260709 13:02:19.561208  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:19.492190 10652 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:19.518584 11148 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.565042  8579 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:19.564566 10652 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.565923 10652 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.568331 11049 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:19.568976 11049 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:19.523375  9327 rpcz_store.cc:269] 0709 13:02:16.936964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.008019 (+ 71055us) service_pool.cc:224] Handling call
0709 13:02:19.492569 (+2484550us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:19.522600 (+ 30031us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13233024}
I20260709 13:02:19.572944 11049 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.562403 11119 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.574581 11119 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.573779  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.576768 10825 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.577092  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.577384 10825 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.578222  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1757) took 2809 ms. Trace:
I20260709 13:02:19.578501  8732 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:19.578948 11057 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.577020  8576 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3. Re-sending same reply.
I20260709 13:02:19.520114 11082 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.574543  8655 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:19.580718  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2073) took 2351 ms. Trace:
I20260709 13:02:19.531899  8724 rpcz_store.cc:276] 0709 13:02:18.463281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.755463 (+292182us) service_pool.cc:224] Handling call
0709 13:02:18.783825 (+ 28362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.785569 (+  1744us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.785595 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:18.799962 (+ 14367us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.953925 (+153963us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:19.031276 (+ 77351us) log.cc:844] Serialized 14 byte log entry
0709 13:02:19.036620 (+  5344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.036743 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.482917 (+446174us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:19.530910 (+ 47993us) raft_consensus.cc:1685] finished
0709 13:02:19.531023 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.531201 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":118159,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":44985,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":17059,"spinlock_wait_cycles":11048704,"thread_start_us":107330,"threads_started":2,"wal-append.queue_time_us":190135}
W20260709 13:02:19.525130  8577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2224) took 2105 ms (client timeout 1523 ms). Trace:
W20260709 13:02:19.551007  8697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.067s	sys 0.064s
W20260709 13:02:19.582787  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 797693us
I20260709 13:02:19.581107  8732 rpcz_store.cc:276] 0709 13:02:17.229176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.229470 (+   294us) service_pool.cc:224] Handling call
0709 13:02:19.578254 (+2348784us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.579157 (+   903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.580517 (+  1360us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b5400339850. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:19.580655 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5360633088}
I20260709 13:02:19.583537  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:19.582080  8577 rpcz_store.cc:269] 0709 13:02:17.419819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.420140 (+   321us) service_pool.cc:224] Handling call
0709 13:02:19.523427 (+2103287us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:19.525025 (+  1598us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5639552}
W20260709 13:02:19.576395  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.507664  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48017.7us
I20260709 13:02:19.558142  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:19.561864  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.590147  8581 raft_consensus.cc:2368] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:19.560235 10917 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:19.577963  8656 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:19.579941 10825 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:19.580152 11148 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:19.599247  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.520007 11035 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.519392 11053 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:19.601567  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.604104  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.604650  8724 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:19.605804  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:19.606225  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.606923  8658 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:19.581046 11082 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:19.578603  8733 rpcz_store.cc:276] 0709 13:02:16.768337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.768993 (+   656us) service_pool.cc:224] Handling call
0709 13:02:16.803675 (+ 34682us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:16.968044 (+164369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:16.968097 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:16.997961 (+ 29864us) log.cc:844] Serialized 26 byte log entry
0709 13:02:17.139245 (+141284us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:17.139298 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.577785 (+2438487us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400339854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:19.577833 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.577876 (+    43us) raft_consensus.cc:1685] finished
0709 13:02:19.577888 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.578146 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22586,"raft.queue_time_us":23142,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":27228,"spinlock_wait_cycles":274281344,"thread_start_us":94316,"threads_started":1,"wal-append.queue_time_us":122444,"wal-append.run_cpu_time_us":1845,"wal-append.run_wall_time_us":1004620}
I20260709 13:02:19.608347  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.609053  8576 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:19.610035  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.610678  8576 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:19.611723  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.612892  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.612944 10917 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:19.614125 10917 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.614708 10825 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:19.614847 11158 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.616156 10825 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:19.616577 10825 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:19.616818 11110 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:19.581461 11118 consensus_queue.cc:1175] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1dc41db601fe4b91aa6d461b504da0e7" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:19.625428  8576 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:19.600085  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.523s (SENT)
W20260709 13:02:19.631194  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45493.1us
I20260709 13:02:19.631388  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.632133  8577 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.633111  8732 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:19.635111  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.637101  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:19.630446  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.644114  8655 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 1.
I20260709 13:02:19.644767  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.645521  8652 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:19.647389  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:19.648036  8733 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:19.649169  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:19.650029  9322 raft_consensus.cc:2368] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6.
I20260709 13:02:19.623286 11098 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:19.651167  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1656) took 1128 ms. Trace:
I20260709 13:02:19.620792 11166 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.652518  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:19.653664 11166 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:19.622225 11016 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 13:02:19.658046 11162 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.617307 11158 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.649852  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:19.651602  8582 rpcz_store.cc:276] 0709 13:02:18.522876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.109158 (+586282us) service_pool.cc:224] Handling call
0709 13:02:19.109582 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.122000 (+ 12418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.122034 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.130501 (+  8467us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.460033 (+329532us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:19.460078 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.460151 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.650960 (+190809us) raft_consensus.cc:1685] finished
0709 13:02:19.650981 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.651110 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1099,"raft.queue_time_us":172720,"raft.run_cpu_time_us":426,"raft.run_wall_time_us":150371,"thread_start_us":297797,"threads_started":1,"wal-append.queue_time_us":483684}
I20260709 13:02:19.646690  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:19.690965  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.658895  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.684177  8651 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:02:19.662743  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:19.697690  9331 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:19.679356  8730 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:19.621807  9325 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 13:02:19.665536  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.701781  8659 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:19.704864  8732 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:19.705118  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.706085  8582 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:19.708986  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 72010.2us
I20260709 13:02:19.710035  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:19.660323  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 99508.1us
I20260709 13:02:19.703701  8581 raft_consensus.cc:2368] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6.
I20260709 13:02:19.617730 10825 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.617456 11110 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.604159  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:19.731600  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.791047  8583 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:19.717906  8482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 304) took 13197 ms (client timeout 15000 ms). Trace:
W20260709 13:02:19.792129  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 82211.2us
I20260709 13:02:19.732537  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1650) took 1234 ms. Trace:
I20260709 13:02:19.716506  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.793108  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:19.730758  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2247) took 2400 ms (client timeout 2034 ms). Trace:
I20260709 13:02:19.734001  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:19.733299  8575 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:19.707806  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.691507 11016 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.794958  8656 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:19.713761  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.795684  8658 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:19.796056  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.713466  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.796733  8656 raft_consensus.cc:2368] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
I20260709 13:02:19.724995  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:19.641796 11053 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.793164  8579 rpcz_store.cc:276] 0709 13:02:18.497876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.052151 (+554275us) service_pool.cc:224] Handling call
0709 13:02:19.732347 (+680196us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54002d73d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:19.732480 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112664832}
I20260709 13:02:19.621212 11160 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:19.798206  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:19.798094  8654 rpcz_store.cc:269] 0709 13:02:17.330459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.360507 (+ 30048us) service_pool.cc:224] Handling call
0709 13:02:19.730555 (+2370048us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54000c80d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:19.730685 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3951033088}
I20260709 13:02:19.797854  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2227) took 2359 ms. Trace:
I20260709 13:02:19.797860  8580 raft_consensus.cc:2376] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4. Re-sending same reply.
I20260709 13:02:19.802771  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.685308 10938 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:19.800401  8575 rpcz_store.cc:276] 0709 13:02:17.438840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.476328 (+ 37488us) service_pool.cc:224] Handling call
0709 13:02:19.732821 (+2256493us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:19.733092 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.794633 (+ 61541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.797790 (+  3157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4678400}
I20260709 13:02:19.813990  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:19.814385  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.815193  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:19.791014  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.702590 11035 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:19.816920  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82981.5us
I20260709 13:02:19.817111  8576 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:19.817975  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:19.823065  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.822221  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:19.724722 11158 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:19.825692  8577 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:19.827898  8579 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:19.828545  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1662) took 1109 ms. Trace:
I20260709 13:02:19.828856  8579 rpcz_store.cc:276] 0709 13:02:18.718926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.798651 (+1079725us) service_pool.cc:224] Handling call
0709 13:02:19.824022 (+ 25371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.828318 (+  4296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.828491 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:19.715214 11119 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:19.832084  8582 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:19.832208  8581 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:19.798108 10652 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:19.646458 10798 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.156s
I20260709 13:02:19.828966  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:19.855201  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.856791  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.857481  8655 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:19.722946  8479 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 4 to 5, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 5 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:02:19.834034  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:19.849750  9296 connection.cc:256] RPC call timeout handler was delayed by -1.8598836209999945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
W20260709 13:02:19.868994  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 515) took 1386 ms (client timeout 1650 ms). Trace:
I20260709 13:02:19.801316 11160 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:19.869870  8727 rpcz_store.cc:269] 0709 13:02:18.482831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.517751 (+ 34920us) service_pool.cc:224] Handling call
0709 13:02:19.868840 (+1351089us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400472cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:19.868939 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":798786304}
I20260709 13:02:19.870319  8657 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:19.855196  8576 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:19.868088  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:19.852736 11035 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:19.872833  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:19.877831  8577 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:19.873250  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.878559  8654 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.872641  8581 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:19.879106  8582 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:19.874951  8650 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:19.880743  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.875663  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:19.880781  8656 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:19.881707  8657 raft_consensus.cc:2368] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:19.867002  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:19.878062  8579 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:19.867883  8725 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:19.891798  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.893081  8656 raft_consensus.cc:2376] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4. Re-sending same reply.
I20260709 13:02:19.893294  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:19.894248  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.894930  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.901685  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:19.906590  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.853495 10811 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.909698  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.802100 11016 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.911407  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.913995  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.918653  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:19.919056  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82697.3us
I20260709 13:02:19.895134  8656 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:19.802307 11053 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:19.859894 11068 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:19.919301  8659 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:19.926429  8550 leader_election.cc:400] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3.
I20260709 13:02:19.914023  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.915982  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.874007 10825 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.891132 10950 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:19.920819  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.877579 11171 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:19.873615 11160 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:19.916473 11158 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:19.872618 11166 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:02:19.860137 11189 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:19.910516  8479 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 3 to 4, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 1 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:19.911043  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44104.7us
I20260709 13:02:19.958328  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:19.926837  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.932047  8727 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:19.960214  8659 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.934628  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:19.920336  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:19.945194  8623 leader_election.cc:400] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6.
I20260709 13:02:19.945353  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:19.947762  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.953171  8656 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:19.867668 11162 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 13:02:19.969951  8732 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:19.961819  8657 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:19.961684  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:19.970649  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:19.971292  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.971583  8733 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:19.961478  8623 leader_election.cc:403] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:19.874902 11111 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 13:02:19.962551  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.973770  8655 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:19.963827  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:19.975184  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.976011  8655 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:19.976107  8730 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:19.978248  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:19.982508  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62229us
I20260709 13:02:19.983474  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:19.987114  8652 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:19.987663  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.989091  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:19.991366  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 162421us
I20260709 13:02:19.925062 11140 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.939081 11134 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:19.995469  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:19.940223 11068 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:19.981379 11043 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:19.948438 11175 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.044107  8655 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:20.025723  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.997265 10839 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 13:02:20.051839 10816 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:02:19.792482  8482 rpcz_store.cc:269] 0709 13:02:06.520316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.524119 (+  3803us) service_pool.cc:224] Handling call
0709 13:02:19.717694 (+13193575us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:15.764622 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:15.765331 (+   709us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:15.765377 (+    46us) write_op.cc:435] Acquired schema lock
0709 13:02:15.765435 (+    58us) tablet.cc:662] Decoding operations
0709 13:02:15.765859 (+   424us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:15.765952 (+    93us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:15.765977 (+    25us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:02:15.766233 (+   256us) tablet.cc:701] Row locks acquired
0709 13:02:15.766282 (+    49us) write_op.cc:260] PREPARE: finished
0709 13:02:15.766590 (+   308us) write_op.cc:270] Start()
0709 13:02:15.766759 (+   169us) write_op.cc:276] Timestamp: P: 1783602135766396 usec, L: 0
0709 13:02:15.766782 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:02:15.767659 (+   877us) log.cc:844] Serialized 530 byte log entry
0709 13:02:16.717703 (+950044us) op_driver.cc:464] REPLICATION: finished
0709 13:02:17.063662 (+345959us) write_op.cc:301] APPLY: starting
0709 13:02:17.063789 (+   127us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:17.063919 (+   130us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:17.064085 (+   166us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:17.064526 (+   441us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:17.064856 (+   330us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:02:17.064914 (+    58us) write_op.cc:312] APPLY: finished
0709 13:02:17.065179 (+   265us) log.cc:844] Serialized 23 byte log entry
0709 13:02:17.080598 (+ 15419us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:17.087971 (+  7373us) write_op.cc:454] Released schema lock
0709 13:02:17.101925 (+ 13954us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:17.174421 (+ 72496us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:17.180076 (+  5655us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1409753,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":332631,"apply.run_cpu_time_us":3577,"apply.run_wall_time_us":74684,"num_ops":1,"prepare.queue_time_us":134365,"prepare.run_cpu_time_us":65662,"prepare.run_wall_time_us":669503,"raft.queue_time_us":187976,"raft.run_cpu_time_us":38538,"raft.run_wall_time_us":213975,"replication_time_us":935499,"thread_start_us":860003,"threads_started":4,"wal-append.queue_time_us":617619,"wal-append.run_cpu_time_us":31591,"wal-append.run_wall_time_us":1721521}]]}
I20260709 13:02:19.970057 11189 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:19.921252 10811 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:19.944311 10950 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:20.058207  8651 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:19.999321  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:20.044898  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:20.078436  8656 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:20.079057  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:20.079658  8657 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:20.058919  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.064344  8658 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:20.020645  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:20.057289  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:20.066538  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82852.1us
I20260709 13:02:19.997702 11123 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:19.983868 11133 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:20.043491 11068 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:20.068512  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 76303.4us
I20260709 13:02:20.082803  8655 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:19.964411 11148 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.118s
I20260709 13:02:20.081789  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:20.121282  8656 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:20.042582 11158 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:20.130301  8658 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:20.118489  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:20.066045 11043 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.143738  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:20.145586  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 288294us
I20260709 13:02:20.148241  8657 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:20.150600  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:20.154335  8696 leader_election.cc:400] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:20.155061  8696 leader_election.cc:403] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:02:20.157759  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:20.161432  8651 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:20.113128 11123 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.160090  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:20.167881  8657 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:20.115762 10938 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.111598 10950 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:20.135223 10839 consensus_queue.cc:1243] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:20.143976 11134 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:20.078127 11189 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:20.130255 10917 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:20.068632 11175 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.046916 11140 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.116714 11133 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:19.998157 11049 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.150s
I20260709 13:02:20.142577 11043 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:20.165654  9327 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:20.116314 10816 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:20.191855  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.161270 11094 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:20.221135 11094 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:20.221849 11094 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:02:20.222177 11198 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 13:02:20.223412 10917 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:20.224439 10917 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.158864 10652 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:20.225595 10652 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:20.226527 10652 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:20.170162  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 88488.9us
I20260709 13:02:20.227538 10816 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.228010  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:20.228708 11200 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:02:20.180131 11202 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.270s	user 0.000s	sys 0.001s
I20260709 13:02:20.229878 11208 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.166677 11191 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:20.229928  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:20.230490 11208 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.231031 11162 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:20.231590 11162 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:20.231722  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:20.232903 11203 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:20.232604 11162 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:20.174842 11110 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:20.170428 11123 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:20.173640 11193 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:20.161685 11173 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:02:20.229178 11200 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:20.207811  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53661.2us
I20260709 13:02:20.263109  8696 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:20.238317  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:20.243714  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:20.251505  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:20.265859  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.162744 11183 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 13:02:20.213748 11133 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:20.260648  8657 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:20.237699 11177 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:20.266567  9327 raft_consensus.cc:2376] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 3. Re-sending same reply.
I20260709 13:02:20.292057  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.292865  9327 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:20.201332 11057 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 13:02:20.293709  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.294421  9327 raft_consensus.cc:2368] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
I20260709 13:02:20.295255  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.295728  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:20.298161 11175 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:20.299300  8657 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 13:02:20.299804  8657 pending_rounds.cc:85] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 5
I20260709 13:02:20.300133  8657 pending_rounds.cc:107] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 13:02:20.300464  8657 raft_consensus.cc:2889] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:20.266438 11200 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:20.301800 11043 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.302748  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.303602  9327 raft_consensus.cc:2376] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4. Re-sending same reply.
I20260709 13:02:20.233510 11203 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.257006 11193 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.287238  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:20.289477 11094 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.296476  9328 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:20.307595  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.310879  9327 raft_consensus.cc:2368] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3.
I20260709 13:02:20.311713  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.302281  8657 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:20.233789 11211 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:20.269488  9322 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:20.188879 10938 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.243640 11049 consensus_queue.cc:1243] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:20.281607  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:20.312528  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.314550  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:20.285969 11134 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:20.305219 11016 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:20.329075 11016 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:20.317384 11212 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:20.329598 11016 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:20.329813 11212 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.330116 11016 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:20.330276 11212 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:20.332881 11208 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:20.338604 11140 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:20.342026 11094 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:20.322861  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:20.265038 11191 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:20.311257 11193 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:20.325489  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:20.326615  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:20.327101 10811 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:20.317799 11053 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:20.346170  8697 connection.cc:256] RPC call timeout handler was delayed by -1.8750235100000054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15180000000000002s for a response
I20260709 13:02:20.319218 11211 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.369554  8651 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:20.372594  9329 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
W20260709 13:02:20.376315  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 416525us
W20260709 13:02:20.376545  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53789.1us
W20260709 13:02:20.324821 11210 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.190s	user 0.002s	sys 0.001s
W20260709 13:02:20.377746  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:20.378073  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 225748us
W20260709 13:02:20.379518  8697 connection.cc:256] RPC call timeout handler was delayed by -1.8516235100000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
W20260709 13:02:20.453285  8697 connection.cc:256] RPC call timeout handler was delayed by -1.6464235099999769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260709 13:02:20.368358  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.324540  9327 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:20.344576 11049 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:20.458469  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:20.269757 11158 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.367148 11208 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.461158  8625 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:20.460453  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 173251us
I20260709 13:02:20.468566  8731 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:02:20.471688  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.462704  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:20.472529  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:20.472859  8577 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:20.473881  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:20.474807  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:20.479354  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:20.483894  8582 raft_consensus.cc:1813] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Handling vote request from an unknown peer 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:20.486586  8582 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:20.483875  8579 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:20.497906  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:20.498862  8729 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:20.499614  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:20.480016  8575 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.004s	sys 0.008s
I20260709 13:02:20.483311  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:20.505266  8575 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.005s	sys 0.009s
I20260709 13:02:20.457832 11053 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:20.513240  8731 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:20.524413  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 366773us
I20260709 13:02:20.467533  8653 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:02:20.533283  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:20.534543  8732 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:20.528332  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:20.485602 11049 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:20.553962  8659 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.001s
W20260709 13:02:20.544418  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45245.2us
I20260709 13:02:20.556154  8659 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:20.556970  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:20.557726  8724 raft_consensus.cc:2376] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 6. Re-sending same reply.
W20260709 13:02:20.543308  8654 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.652s	user 0.010s	sys 0.001s
I20260709 13:02:20.560703  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:20.550123  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:20.529366 11207 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:20.459649 11211 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:20.575387  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:20.525563 11208 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.584117 10938 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:20.584945 11203 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:20.586961 10825 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.640s	user 0.007s	sys 0.003s
I20260709 13:02:20.559340 11082 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6.
I20260709 13:02:20.559669  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:20.560087 11049 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:20.537959 11043 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:20.591581  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1657) took 2060 ms. Trace:
I20260709 13:02:20.591830 11230 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:20.599813  8654 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:20.679680 11209 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:20.680418 11209 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.680897 11209 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:20.683499  8725 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
W20260709 13:02:20.602355  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 358616us
W20260709 13:02:20.530561 11035 consensus_meta.cc:287] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.642s	user 0.011s	sys 0.001s
I20260709 13:02:20.594626  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2249) took 1127 ms. Trace:
I20260709 13:02:20.610775  8728 raft_consensus.cc:2410] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:20.582545  9328 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:20.531227 11212 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:20.615446  9327 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:20.618572  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:20.641624  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:20.589335 11160 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.594s	user 0.007s	sys 0.000s
I20260709 13:02:20.589017 11200 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.667030  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:20.674021  8578 rpcz_store.cc:276] 0709 13:02:18.531587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.357876 (+826289us) service_pool.cc:224] Handling call
0709 13:02:19.374040 (+ 16164us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.787584 (+413544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.787617 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.800134 (+ 12517us) log.cc:844] Serialized 26 byte log entry
0709 13:02:19.929161 (+129027us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:19.966459 (+ 37298us) log.cc:844] Serialized 14 byte log entry
0709 13:02:19.993245 (+ 26786us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.993324 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.301020 (+307696us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:20.591303 (+290283us) raft_consensus.cc:1685] finished
0709 13:02:20.591335 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.591318 (+   -17us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b18003152e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:20.591506 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39677,"prepare.queue_time_us":37069,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":93875,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":55800,"spinlock_wait_cycles":83307520,"thread_start_us":389444,"threads_started":2,"wal-append.queue_time_us":103681}
I20260709 13:02:20.598484  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:20.593360 11054 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260709 13:02:20.571895 11214 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
I20260709 13:02:20.571025 11220 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.181s
I20260709 13:02:20.685325  8584 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:20.879027  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:21.108266 11123 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.850s	user 0.010s	sys 0.001s
I20260709 13:02:21.161962  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2277) took 1494 ms. Trace:
I20260709 13:02:21.162170 11160 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.162339 11123 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.162612  8650 rpcz_store.cc:276] 0709 13:02:19.668097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.880341 (+212244us) service_pool.cc:224] Handling call
0709 13:02:20.598236 (+717895us) spinlock_profiling.cc:243] Waited 890 us on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:21.161901 (+563665us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2046848}
I20260709 13:02:21.163447  8729 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:20.792824 11068 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.647s	user 0.010s	sys 0.000s
I20260709 13:02:20.784200 11211 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:21.163661 11222 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.164162  8657 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:21.164095 11068 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:21.164861  8731 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:21.164575 11222 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:21.166177  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.166954  8731 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:21.167241 11222 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:21.102851 11189 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.885s	user 0.007s	sys 0.004s
I20260709 13:02:20.738897 11230 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:21.168660 11189 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:21.169346 11230 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:21.132246 11193 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.683s	user 0.006s	sys 0.005s
I20260709 13:02:21.172272 11193 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.592093 11216 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:21.174003  8724 raft_consensus.cc:2376] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 5. Re-sending same reply.
I20260709 13:02:21.009661  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:21.112345  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.030s	sys 0.029s
W20260709 13:02:21.177412  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.031s	sys 0.030s
I20260709 13:02:21.177745  8725 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:20.791968 10950 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.587s	user 0.007s	sys 0.000s
I20260709 13:02:21.110780 11209 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.223194  9329 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:21.161988  8582 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:21.185374 11035 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:21.226996 11209 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:21.138266 11016 consensus_meta.cc:287] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.807s	user 0.011s	sys 0.001s
I20260709 13:02:21.179100  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:21.227972 11016 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:21.177796  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 699255us
I20260709 13:02:21.231468 11016 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:20.782847  9323 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:21.232841  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 696) took 1005 ms. Trace:
I20260709 13:02:21.233325  9323 rpcz_store.cc:276] 0709 13:02:20.227441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.782013 (+554572us) service_pool.cc:224] Handling call
0709 13:02:20.782432 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.232394 (+449962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.232756 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:20.794075 11171 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:21.234822 11171 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:21.236045 11171 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.678835 11133 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:20.786315 11166 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.237920 11166 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.240007 11171 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:20.687016 10938 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.507s
I20260709 13:02:21.166975 11068 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:20.735750 11213 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
I20260709 13:02:21.136210  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:21.100739  8699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.012s	sys 0.021s
W20260709 13:02:21.246057  8699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.013s	sys 0.021s
W20260709 13:02:21.246632  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.246783  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1664) took 2495 ms. Trace:
I20260709 13:02:21.247218  8583 rpcz_store.cc:276] 0709 13:02:18.750981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.803974 (+1052993us) service_pool.cc:224] Handling call
0709 13:02:19.834806 (+ 30832us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.835207 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.835237 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.835714 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.171118 (+335404us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:20.171209 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.171336 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.246540 (+1075204us) raft_consensus.cc:1685] finished
0709 13:02:21.246563 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.246722 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18384,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":6353,"thread_start_us":249971,"threads_started":1,"wal-append.queue_time_us":567476}
I20260709 13:02:20.844133 11053 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:20.798215 11175 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:21.250136 11175 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:21.250679 11053 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:21.251170 11175 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.252338 11175 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:21.010128  8579 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:21.210634  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 371) took 1468 ms. Trace:
I20260709 13:02:21.262593  8658 rpcz_store.cc:276] 0709 13:02:19.742623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.136186 (+393563us) service_pool.cc:224] Handling call
0709 13:02:20.136577 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:20.136996 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:20.137024 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:20.137524 (+   500us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.477733 (+340209us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:20.477772 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.477847 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.682031 (+204184us) raft_consensus.cc:1685] finished
0709 13:02:20.682051 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.210528 (+528477us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":14547,"thread_start_us":238227,"threads_started":1,"wal-append.queue_time_us":475218}
I20260709 13:02:20.777640 11226 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:21.222342 10950 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:20.734826 11215 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 13:02:21.113777  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.175246  8575 rpcz_store.cc:276] 0709 13:02:19.466926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.823117 (+356191us) service_pool.cc:224] Handling call
0709 13:02:19.823423 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.823751 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.823793 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.877324 (+ 53531us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.507100 (+629776us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:20.507177 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.507549 (+   372us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.594385 (+ 86836us) raft_consensus.cc:1685] finished
0709 13:02:20.594406 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.594555 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":567965,"threads_started":1,"wal-append.queue_time_us":679237}
I20260709 13:02:21.240208 11166 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:20.824283  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.283049  8733 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:21.283236  8655 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:21.280395  8732 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:21.157950  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2287) took 1281 ms. Trace:
I20260709 13:02:20.779733 11156 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:20.658550 11173 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:20.869307 11043 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.160640 11049 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:20.781309 11204 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.638s	user 0.001s	sys 0.002s
W20260709 13:02:21.177927 11191 consensus_meta.cc:287] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.800s	user 0.005s	sys 0.007s
I20260709 13:02:21.171298 11189 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:21.283910  8656 rpcz_store.cc:276] 0709 13:02:19.876685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.206956 (+330271us) service_pool.cc:224] Handling call
0709 13:02:20.207326 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:20.207692 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:20.207718 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:20.213089 (+  5371us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.486524 (+273435us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:20.486574 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.486643 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.157684 (+671041us) raft_consensus.cc:1685] finished
0709 13:02:21.157705 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.157877 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":23810,"thread_start_us":265242,"threads_started":1,"wal-append.queue_time_us":459154}
W20260709 13:02:21.291007  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 112020us
I20260709 13:02:21.366207  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:21.259276  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 690) took 1080 ms. Trace:
I20260709 13:02:21.160426 10652 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:21.282900  9297 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.016s	sys 0.019s
W20260709 13:02:21.370048  9297 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.016s	sys 0.019s
W20260709 13:02:21.370488  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 825339us
I20260709 13:02:21.127317 11208 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:21.306567  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.002s	user 0.005s	sys 0.032s
W20260709 13:02:21.379081  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.075s	user 0.006s	sys 0.032s
I20260709 13:02:21.216717 11216 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:21.364203  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:21.364794  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 104186us
I20260709 13:02:21.282440 11160 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:21.384363  8698 connection.cc:256] RPC call timeout handler was delayed by -1.2517040469999756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260709 13:02:21.368130  9330 rpcz_store.cc:276] 0709 13:02:20.178765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.178941 (+   176us) service_pool.cc:224] Handling call
0709 13:02:20.179295 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:20.179540 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:20.476235 (+296695us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.258833 (+782598us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:21.258892 (+    59us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:21.258953 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.259027 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.259172 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":358309,"prepare.queue_time_us":395021,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":368525,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":9968,"thread_start_us":711109,"threads_started":2}
I20260709 13:02:21.387881  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:21.301797 11173 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.290691 11156 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:21.389111  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:21.391300  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:21.373811 11234 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 13:02:21.410849 11222 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.411631  8651 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:21.411773 11189 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:21.412184 11189 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.412568 11189 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:21.413707 11185 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:21.414151 11185 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:21.415076 11185 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:21.411556  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:21.416882  8650 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:21.394021  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:21.395437  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:21.418114  8654 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:21.320782 11206 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.000s	sys 0.013s
I20260709 13:02:21.323997 11098 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:21.401129  8698 connection.cc:256] RPC call timeout handler was delayed by -1.1168578240000215s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20260709 13:02:21.432916  8698 connection.cc:256] RPC call timeout handler was delayed by -1.0790900900000224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
I20260709 13:02:21.403707  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:21.418009  8652 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:21.433331  8698 connection.cc:256] RPC call timeout handler was delayed by -1.068104046999963s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
I20260709 13:02:21.396093  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:21.364533 11227 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:02:21.421952  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.378341 11035 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.378327 11191 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:21.418845  8658 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:21.355123 11043 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.419430 11173 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:21.423009 11206 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.000s	sys 0.014s
I20260709 13:02:21.462412  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:21.452283  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48667.1us
I20260709 13:02:21.465427  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.465641  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:21.452586  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:21.291276 11226 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.466143  8726 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:21.466511  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:21.456128  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 225899us
I20260709 13:02:21.399044 11220 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:21.182085 11212 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:21.459911  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.466315  8730 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:21.467682  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:21.480786  8732 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:21.468297  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1663) took 2728 ms. Trace:
I20260709 13:02:21.481575  8580 rpcz_store.cc:276] 0709 13:02:18.739653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.803678 (+1064025us) service_pool.cc:224] Handling call
0709 13:02:19.855621 (+ 51943us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.856146 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.856190 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:19.884514 (+ 28324us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.268085 (+383571us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:20.268151 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.268259 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.468094 (+1199835us) raft_consensus.cc:1685] finished
0709 13:02:21.468115 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.468246 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13089,"raft.queue_time_us":13152,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":11102,"thread_start_us":325198,"threads_started":1,"wal-append.queue_time_us":545804}
I20260709 13:02:21.390998 11016 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260709 13:02:21.472674  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:21.485244  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:21.485567  9329 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:21.474436  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.475252  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.472715  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.477133 11226 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:21.477645 11216 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:21.458190  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.468466 11227 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:21.517565 11226 raft_consensus.cc:2764] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:02:21.527244 11227 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:21.486181  8728 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:21.530042 11200 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:21.530284  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.517563 11160 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:21.514876 11206 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:21.523911 11175 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 13:02:21.532039 11206 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.516587  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:21.487385  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.453972  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:21.533485  9325 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:21.534018  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 695) took 1307 ms. Trace:
I20260709 13:02:21.496743  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:21.534546 11206 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:21.460434 11095 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:02:21.505650  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53131.7us
I20260709 13:02:21.536787  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.537863  8732 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:21.508131  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.538015  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:21.468143 11207 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.411592 11222 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.483418  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:21.510128 11220 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:21.518543 10950 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:21.597033  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 129442us
I20260709 13:02:21.521620 11183 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:21.522378  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.525473 11043 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:21.463968 11035 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.526135 11193 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:21.531493 11160 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.533103  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 499) took 1940 ms. Trace:
I20260709 13:02:21.502635  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.613061  8724 raft_consensus.cc:2376] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4. Re-sending same reply.
W20260709 13:02:21.613181  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.537906  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.455377 11098 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.543844  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.583890  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.615041  8731 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 1.
I20260709 13:02:21.611986  8584 rpcz_store.cc:276] 0709 13:02:19.592716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.819280 (+226564us) service_pool.cc:224] Handling call
0709 13:02:19.869999 (+ 50719us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.161424 (+1291425us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.532830 (+371406us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b5400343ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:21.533035 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1850021248}
W20260709 13:02:21.596453  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58516.5us
I20260709 13:02:21.519181 11123 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:21.583374  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.596992  9329 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:21.603372 11016 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:21.597321 11207 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.534366  9325 rpcz_store.cc:276] 0709 13:02:20.227070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.683005 (+455935us) service_pool.cc:224] Handling call
0709 13:02:21.516396 (+833391us) spinlock_profiling.cc:243] Waited 80.8 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:21.533955 (+ 17559us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185743104}
I20260709 13:02:21.628577  8733 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:21.603758  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 216945us
I20260709 13:02:21.599110 11049 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:21.635460  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 124957us
I20260709 13:02:21.636107  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:21.637580  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 127955us
I20260709 13:02:21.606598 11247 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.636724  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:21.614776 11160 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:21.631883  8732 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:21.630432  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.630795  8622 leader_election.cc:400] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6.
I20260709 13:02:21.633018  8728 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:21.641661  9330 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
W20260709 13:02:21.627808  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.634421  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:21.635267  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104717us
I20260709 13:02:21.642853  9329 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:21.642292  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:21.644080  8698 leader_election.cc:400] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
W20260709 13:02:21.644098  8549 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.660s (SENT)
I20260709 13:02:21.644577  8698 leader_election.cc:403] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 13:02:21.633036  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 257039us
W20260709 13:02:21.646884  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.647413  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:21.647729  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:21.489619 11156 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:21.654027  9297 connection.cc:256] RPC call timeout handler was delayed by -1.1598415060000207s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260709 13:02:21.646023  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:21.654498  9297 connection.cc:256] RPC call timeout handler was delayed by -1.0401415060000545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260709 13:02:21.629714 11095 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:21.631326 11016 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:21.661031  8697 leader_election.cc:400] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
I20260709 13:02:21.656682  8575 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:21.658071  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:21.661839  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:21.614893  8725 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:21.663050  8584 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:21.663868  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:21.671525  9329 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:21.684814  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48388.8us
I20260709 13:02:21.687806 11123 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:21.685781  8578 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:21.692862  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2105) took 1186 ms. Trace:
I20260709 13:02:21.698078  8729 rpcz_store.cc:276] 0709 13:02:20.507022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.529129 (+ 22107us) service_pool.cc:224] Handling call
0709 13:02:20.557205 (+ 28076us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.163859 (+606654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.692584 (+528725us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b540022a954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:21.692793 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393564032}
I20260709 13:02:21.689810 11123 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:21.701548 11250 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:21.702041  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.702877  9331 raft_consensus.cc:2376] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 3. Re-sending same reply.
I20260709 13:02:21.703801  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 687) took 1543 ms. Trace:
I20260709 13:02:21.690101  8732 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:02:21.704160  9331 rpcz_store.cc:276] 0709 13:02:20.163825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.168803 (+  4978us) service_pool.cc:224] Handling call
0709 13:02:21.701853 (+1533050us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:21.703450 (+  1597us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2995456}
I20260709 13:02:21.706144 11185 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:21.668165  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:21.708082 11162 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:21.709951  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:21.716652  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.717566  8733 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:21.718544  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.720706  9331 raft_consensus.cc:2368] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3.
I20260709 13:02:21.721746  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.716548  8731 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:21.714588  8732 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:21.728943  9331 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:21.730036  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:21.730679  9331 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:21.733124  8732 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:21.852511  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.665256 11016 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:21.657804 11226 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:21.695240 11228 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.974s	user 0.008s	sys 0.004s
W20260709 13:02:21.980299  9327 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 530) took 1694 ms (client timeout 2041 ms). Trace:
W20260709 13:02:22.010723  9327 rpcz_store.cc:269] 0709 13:02:20.285516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.311464 (+ 25948us) service_pool.cc:224] Handling call
0709 13:02:21.980089 (+1668625us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5400816354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:21.980238 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25348352}
I20260709 13:02:21.696334 11054 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
I20260709 13:02:21.696833 11156 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:21.981774  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 372) took 2238 ms. Trace:
I20260709 13:02:21.742281 11200 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:22.030886  8726 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:21.985670  9331 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:21.683466 11068 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:22.022322  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 360214us
I20260709 13:02:22.022548  8655 rpcz_store.cc:276] 0709 13:02:19.742898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.140411 (+397513us) service_pool.cc:224] Handling call
0709 13:02:20.140835 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.290356 (+1149521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.981526 (+691170us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400059bd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:21.981709 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2626333952}
I20260709 13:02:21.652969 11183 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:22.033196  8697 leader_election.cc:400] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3.
W20260709 13:02:21.983234  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:22.035583 11231 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.257s	user 0.002s	sys 0.013s
W20260709 13:02:22.036059 11199 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.967s	user 0.013s	sys 0.005s
I20260709 13:02:22.036424 11239 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.037052 11239 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:22.017979 11242 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:02:21.985050 11250 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:22.055517  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:21.975956 11175 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260709 13:02:22.059556  8623 leader_election.cc:400] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3.
I20260709 13:02:22.059911  8623 leader_election.cc:403] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260709 13:02:22.061589  9297 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.536s (SENT)
I20260709 13:02:22.062258  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:22.038931 11200 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:22.066196  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:22.093614  9297 leader_election.cc:336] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.669s (SENT)
W20260709 13:02:22.095156  9297 leader_election.cc:336] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.774s (SENT)
I20260709 13:02:22.095757  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.057962  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:21.653334 11247 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:22.053184 11171 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:22.049911  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 314286us
W20260709 13:02:22.099944  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 441866us
W20260709 13:02:22.109970  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 52114.5us
I20260709 13:02:22.111027  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:22.099005  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:22.112828  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 53404.8us
I20260709 13:02:22.022557 11226 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:21.978735  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.102815  9329 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:22.102047  8578 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:22.111107  8698 leader_election.cc:400] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:22.057933 11250 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.121879 11260 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:22.062732 10652 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.684s	user 0.016s	sys 0.004s
I20260709 13:02:22.123101  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:22.123489  9297 leader_election.cc:336] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.947s (SENT)
I20260709 13:02:22.131565  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.041280 11043 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 13:02:22.135438  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:22.087548  9325 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:22.056572 11183 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:22.124130  8697 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:22.105857 11189 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.693s	user 0.011s	sys 0.011s
I20260709 13:02:22.141808  8727 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:22.086917 11252 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.172s
I20260709 13:02:22.104765  8725 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:22.049968 11068 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:21.737926 11220 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:22.148109  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.098843 11200 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:22.038218 11095 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:22.199740  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51555.7us
I20260709 13:02:22.087320 11251 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:22.104698 10917 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:22.175158  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:22.186153  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45569.4us
W20260709 13:02:22.189109  8550 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.548s (SENT)
I20260709 13:02:22.267377  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.200280  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:22.200938  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:22.268550  8579 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:22.147758  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.134804 11238 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:22.271376  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:22.201347  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:22.274684  8584 raft_consensus.cc:2368] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3.
I20260709 13:02:22.267342  8581 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:22.202517  8730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.016s	sys 0.003s
I20260709 13:02:22.202764  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:22.275556  8730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.017s	sys 0.003s
I20260709 13:02:22.275702  8578 raft_consensus.cc:1813] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Handling vote request from an unknown peer d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.276628  8578 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:22.177492  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:22.132715 11227 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:22.216318  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:22.134989 11054 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:22.147342 11222 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:22.339043  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:22.222478  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:22.267627  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.175689  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:22.140252 11234 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:22.129953  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:22.146294 11206 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:22.368119  9329 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:22.185930 11162 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:22.126170 11171 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:22.313498 11245 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 13:02:22.217751 11193 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:22.388459  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:22.399606 11248 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:22.400257 11261 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.223963 11098 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:22.372473 11035 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:22.186254 11189 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:22.498208  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.145962 11260 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:22.410933  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:22.511377  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:22.441834  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:22.445864  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:22.513561  8658 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:22.514009  8731 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:22.229471 11253 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:22.401441  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 688) took 2236 ms (client timeout 1996 ms). Trace:
I20260709 13:02:22.490556  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:22.377978 10917 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:22.506392  8581 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:22.507747  8656 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:02:22.531965  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.407075 11193 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:22.622289  9324 rpcz_store.cc:269] 0709 13:02:20.164511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.165236 (+   725us) service_pool.cc:224] Handling call
0709 13:02:22.122861 (+1957625us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:22.401292 (+278431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32792960}
W20260709 13:02:22.644032  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.026s	sys 0.045s
W20260709 13:02:22.644452  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.026s	sys 0.045s
W20260709 13:02:22.647069  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 511604us
I20260709 13:02:22.488582  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:22.492664 11098 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:22.661505  8656 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:22.384531 11234 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.374557 11238 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.663748  8583 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:22.401888 11206 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.618216  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:22.678052  8725 raft_consensus.cc:2376] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4. Re-sending same reply.
W20260709 13:02:22.687132  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 419474us
I20260709 13:02:22.688010  8698 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.657188 11123 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.694618  8726 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:22.675786  9329 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:02:22.694069  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:22.696967 11275 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:22.698930 11049 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.699339 11260 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:22.699887 11250 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:22.687570 11240 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.534s	user 0.000s	sys 0.001s
I20260709 13:02:22.683723 11249 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:22.699389 11049 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:22.692184 10652 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:22.703226  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:22.703830  8732 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:22.704808 11049 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:22.627146 10917 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.670842 11236 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:22.705775 10917 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:22.626267 11183 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.707346  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.708690  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:22.709017 11171 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:22.709216 11280 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:22.709901 11271 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:22.710289 11266 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 13:02:22.710516 11271 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.711097 11271 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:22.681077 11264 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:22.712832 11226 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:22.714534  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:22.716179  8656 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
W20260709 13:02:22.649276 11252 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.068s	sys 0.081s
I20260709 13:02:22.627254  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:22.685887  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 496675us
W20260709 13:02:22.686200  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:22.718492  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.008s	sys 0.001s
I20260709 13:02:22.620613 11035 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:22.663677 11043 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:22.679091  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:22.697342 11259 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.590s	user 0.000s	sys 0.001s
I20260709 13:02:22.706244 11236 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:22.679342 11173 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.193s	user 0.010s	sys 0.000s
I20260709 13:02:22.641939 11193 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:22.622474 11222 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:22.662441  9324 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 13:02:22.719028  8550 connection.cc:256] RPC call timeout handler was delayed by -1.0226389040000186s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
W20260709 13:02:22.771711  8550 connection.cc:256] RPC call timeout handler was delayed by -1.0001379789999874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
W20260709 13:02:22.718801  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.008s	sys 0.001s
W20260709 13:02:22.730688  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 529678us
I20260709 13:02:22.491787 11261 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:22.711159 11068 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:22.728915 11156 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.702s	user 0.002s	sys 0.007s
I20260709 13:02:22.752509  9329 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:22.710196 11280 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:02:22.718327 11252 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.068s	sys 0.082s
I20260709 13:02:22.689131 11239 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:22.710156  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:22.685652 11098 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:22.788151  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.946858  8732 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:22.773451  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 650257us
W20260709 13:02:22.930635  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.987262  8581 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:22.988692  8575 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 13:02:22.992215  8575 pending_rounds.cc:85] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 2
I20260709 13:02:22.992100  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:22.992619  8575 pending_rounds.cc:107] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 13:02:22.992871  8575 raft_consensus.cc:2889] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:22.773085  8656 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 13:02:22.977180  8730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.002s	sys 0.016s
W20260709 13:02:22.994438  8730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.002s	sys 0.016s
W20260709 13:02:22.981292  8549 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.004s	sys 0.039s
W20260709 13:02:22.976929  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.999724  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.000319  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:23.000965  8654 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:23.000860  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 290712us
I20260709 13:02:23.001456  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.001755  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:23.003239  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:22.746879 11236 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:23.005728  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.007457  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:22.945354 10938 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:22.817731 11254 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.010543  8579 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:23.010485  8576 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:22.769552  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:22.719345 11123 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.866520 11054 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:22.928905 11207 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:22.943080 11234 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:22.769101 11043 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.982523 11280 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:23.061738 11280 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:22.944658 11189 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:23.011140  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.037799 11254 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:22.759871 11173 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.048341  8654 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:23.042917 11156 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.044134  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:23.009874 11068 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:23.011405 11098 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:22.983355 11273 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:23.029085 11016 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.007s	user 0.009s	sys 0.004s
I20260709 13:02:23.092427 11016 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.056164  8576 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:23.013741 10938 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:23.009582  8549 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.004s	sys 0.039s
I20260709 13:02:23.094390  9322 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:23.094956  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 917331us
I20260709 13:02:23.095216  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 701) took 2866 ms. Trace:
I20260709 13:02:23.095940  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:23.058748  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:23.098575  9329 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:23.020496 11171 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:23.102178  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:23.070832  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.103582  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:23.094341 10938 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.073004  8732 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:23.107854  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.108645  8732 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:23.105078  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:23.075078 11269 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:23.048465  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:23.073411 11270 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 13:02:23.047859 11252 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.029346 11049 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 13:02:23.088080 11273 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.030705 11035 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:02:23.100615  8731 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:23.055994  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:23.065637 11279 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.102329 11171 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.078864  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:23.111488  9297 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:23.113298  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.131275  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.132814  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.133544  8732 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
W20260709 13:02:23.117416  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68988.6us
I20260709 13:02:23.137008  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:23.088860  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:23.115924  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:23.123731 11162 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.717s	user 0.010s	sys 0.000s
I20260709 13:02:23.057072 11183 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20260709 13:02:23.125715  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69780.1us
I20260709 13:02:23.088188  8725 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:23.135777  8732 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:23.070675 11206 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.154145  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:23.113083 11276 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:23.149976  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:23.160188  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 143979us
I20260709 13:02:23.163508  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:23.046844 11282 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 13:02:23.139015 11171 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:23.106956 11148 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 13:02:23.157215 11260 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:23.169792  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.138630 11068 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:23.089274 11156 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.095691  9322 rpcz_store.cc:276] 0709 13:02:20.229193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.794112 (+564919us) service_pool.cc:224] Handling call
0709 13:02:23.062954 (+2268842us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b54005178a8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140ebbaebc 00007f140ebf008c 00007f140ebda93b 00007f140ebda3d2 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c
0709 13:02:23.079636 (+ 16682us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.094934 (+ 15298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.095153 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23341312}
I20260709 13:02:23.157617 11098 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.208110  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:23.273079  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65060us
I20260709 13:02:23.121584 11222 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260709 13:02:23.231909 11265 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 13:02:23.240957 11276 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:23.121778 11252 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.242280 11281 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:23.243896  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:23.291069  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47219.8us
W20260709 13:02:23.244433  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 94575.8us
W20260709 13:02:23.259595  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 243145us
I20260709 13:02:23.160120 11162 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:23.262643  8576 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:23.107353 11253 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260709 13:02:23.159595 10839 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:23.164276 11140 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:23.277041  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2145) took 3292 ms. Trace:
I20260709 13:02:23.278759 11282 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:23.302199  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:23.309813  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:23.327469  8584 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:23.165938 11094 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
I20260709 13:02:23.320490  8657 rpcz_store.cc:276] 0709 13:02:19.972237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.313182 (+340945us) service_pool.cc:224] Handling call
0709 13:02:20.313476 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.164868 (+851392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.264557 (+2099689us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b5400420c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:23.264822 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1955612544}
I20260709 13:02:23.293931  8622 leader_election.cc:400] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 2 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3.
I20260709 13:02:23.332461  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:23.264295 11290 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:23.330353  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 166924us
I20260709 13:02:23.338300  8550 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:23.282682 11035 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:23.264088 11234 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
W20260709 13:02:23.338631  8655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.023s	sys 0.063s
I20260709 13:02:23.340595  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:23.309095 11250 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:23.317489 11222 consensus_queue.cc:1243] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:02:23.369082  8549 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.825s (SENT) [suppressed 59 similar messages]
W20260709 13:02:23.411755  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71305.7us
I20260709 13:02:23.414250  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:23.414819  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45771.3us
I20260709 13:02:23.341264  8732 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:23.259526 11068 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.320443 11140 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:23.320114 10839 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:23.436846  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 134720us
I20260709 13:02:23.437059 10839 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:23.438021 11016 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:23.439363 10917 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.733s	user 0.012s	sys 0.000s
I20260709 13:02:23.439790 10917 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.340065 11035 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.440850 11035 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:23.441649 11291 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:23.442102 11291 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.442153 11270 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.442521 11291 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:23.442741 11270 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.326242 11276 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:23.349354 11160 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.693s	user 0.012s	sys 0.002s
W20260709 13:02:23.362529 11268 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 13:02:23.315790 11283 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:23.417912  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 537) took 1146 ms. Trace:
I20260709 13:02:23.337847 11290 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.460988 11290 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:23.460691 11160 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.165171 11279 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.463070  8575 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:23.460683  8732 rpcz_store.cc:276] 0709 13:02:22.271949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.268647 (+996698us) service_pool.cc:224] Handling call
0709 13:02:23.314686 (+ 46039us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.417632 (+102946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.417854 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:23.325253 11282 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:23.419706  8584 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:23.434518  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:23.468770  8548 connection.cc:256] RPC call timeout handler was delayed by -1.6011059189999628s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2542s for a response
I20260709 13:02:23.436864 11140 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:23.436218 11156 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.391127 11250 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.281258 11054 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.426471 11068 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:23.440227 11173 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.280787 11098 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.470121 11082 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:23.467031  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 134650us
I20260709 13:02:23.452904 11148 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:02:23.470616  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:23.413873 11123 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.473153  8581 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:23.476416 11254 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.143215 11269 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.442163 10917 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.430809 11298 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:23.485479 11250 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:23.486222 11193 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.657s	user 0.005s	sys 0.006s
I20260709 13:02:23.493458  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:23.439659 11287 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 13:02:23.436101  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:23.488828 11227 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.124s	user 0.009s	sys 0.005s
I20260709 13:02:23.507333  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:23.535462  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 99413.8us
I20260709 13:02:23.536221 11123 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:23.536268  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:23.536996 11123 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:23.537425 11173 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:23.538187  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:23.538511 11123 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:23.511801  9326 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 691) took 3332 ms (client timeout 2113 ms). Trace:
I20260709 13:02:23.494426  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:23.539758 11222 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:23.540052  9322 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:23.540360 11222 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:23.540925 11222 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:23.539724  9326 rpcz_store.cc:269] 0709 13:02:20.179149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.200329 (+ 21180us) service_pool.cc:224] Handling call
0709 13:02:20.898717 (+698388us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:23.470409 (+2571692us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1e9c 00007f1406af1a5b 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:23.511693 (+ 41284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84957184}
I20260709 13:02:23.539265 11297 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:23.506734 11264 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:23.509223 11253 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:23.552033 11252 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.552774 11156 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:23.510949 11283 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:23.523763 11200 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.250s	user 0.006s	sys 0.008s
I20260709 13:02:23.508685 11273 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.514171 11298 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.571365  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2269) took 1924 ms. Trace:
I20260709 13:02:23.571758  8577 rpcz_store.cc:276] 0709 13:02:21.646641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.646768 (+   127us) service_pool.cc:224] Handling call
0709 13:02:21.658065 (+ 11297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.986570 (+328505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.986614 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.005572 (+ 18958us) log.cc:844] Serialized 26 byte log entry
0709 13:02:22.526700 (+521128us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:22.526770 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.526869 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.571163 (+1044294us) raft_consensus.cc:1685] finished
0709 13:02:23.571182 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.571308 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":397888,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":54071,"thread_start_us":798508,"threads_started":2,"wal-append.queue_time_us":842107}
I20260709 13:02:23.575090  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.566740 11279 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.553146 11227 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3.
I20260709 13:02:23.583804 11156 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:23.511524 11082 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.588971  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:23.596084  8653 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:23.604566 11290 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.549439 11193 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.578390 11298 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:23.534965 11276 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.613906 11156 raft_consensus.cc:2764] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 13:02:23.613139  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:23.555195 11282 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.618028 11282 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:23.619233  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:23.555625 11280 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.570492 11200 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.627334  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:23.642699  8548 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.153s (SENT)
I20260709 13:02:23.635281  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:23.649753 11300 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:23.650421 11300 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:23.650971 11300 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:23.610726 11160 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:23.659780  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:23.621649 11279 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:23.666306  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:23.608776 10839 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.622895 11035 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.667728 11274 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:23.668058 11098 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.642298 11156 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
W20260709 13:02:23.627807 11236 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.607s	user 0.012s	sys 0.000s
I20260709 13:02:23.674788  8659 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:23.619305 11254 consensus_queue.cc:1175] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0d703ee18464475a290425976246555" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 4 index: 3 } }
I20260709 13:02:23.677861 11297 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:23.678735 11133 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.621553 11270 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.688066 11281 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.674497 11252 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:23.619048 11291 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.674006 11279 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:23.703810  8732 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:23.702262 11281 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.711153  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2124) took 1410 ms. Trace:
I20260709 13:02:23.713116  8732 rpcz_store.cc:276] 0709 13:02:22.301112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.674160 (+1373048us) service_pool.cc:224] Handling call
0709 13:02:23.701337 (+ 27177us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.710777 (+  9440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.711072 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:23.713281  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:23.725015  8583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1678) took 1586 ms (client timeout 1793 ms). Trace:
W20260709 13:02:23.725724  8583 rpcz_store.cc:269] 0709 13:02:22.139063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.187104 (+ 48041us) service_pool.cc:224] Handling call
0709 13:02:23.724740 (+1537636us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54003e7454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:23.724922 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682439424}
I20260709 13:02:23.706998 10839 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.732648  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2298) took 2335 ms. Trace:
I20260709 13:02:23.733362  8652 rpcz_store.cc:276] 0709 13:02:21.397368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.433874 (+ 36506us) service_pool.cc:224] Handling call
0709 13:02:23.713054 (+2279180us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:23.732563 (+ 19509us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46076672}
I20260709 13:02:23.703642 11252 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.704487 11035 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:23.702750 11297 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:23.732535 11269 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.708216 11279 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:23.737108 11193 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.746654  8732 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:23.758414  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2126) took 1456 ms. Trace:
I20260709 13:02:23.729755 11290 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.701586 11222 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.758973  8732 rpcz_store.cc:276] 0709 13:02:22.301733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.714857 (+1413124us) service_pool.cc:224] Handling call
0709 13:02:23.746317 (+ 31460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.757986 (+ 11669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.758327 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:23.698505 11283 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.762126  8732 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:23.765424  8732 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:23.742676 11252 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:23.793830 11292 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:23.794498 11292 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:23.745163 11260 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:23.696084 11236 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:23.766379 11280 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.804693 11292 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.824438 11292 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:23.827709 11238 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:23.846144  8732 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:23.862112  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 711) took 2205 ms. Trace:
I20260709 13:02:23.862679  9331 rpcz_store.cc:276] 0709 13:02:21.657066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.731336 (+ 74270us) service_pool.cc:224] Handling call
0709 13:02:21.985418 (+254082us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.034569 (+ 49151us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.861875 (+1827306us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400816350. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:23.862040 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":583247744}
I20260709 13:02:23.866003  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:23.855484 11200 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.858326  8733 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:23.874387  8733 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:23.874482  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:23.875150  8733 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.875789  8581 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:23.981880  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:23.878536  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2099) took 3406 ms (client timeout 1597 ms). Trace:
W20260709 13:02:23.893355  9294 connection.cc:256] RPC call timeout handler was delayed by -1.1301357549999693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25s for a response
W20260709 13:02:23.984107  8727 rpcz_store.cc:269] 0709 13:02:20.454301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.454495 (+   194us) service_pool.cc:224] Handling call
0709 13:02:23.859968 (+3405473us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54004731d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:23.860083 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3811349760}
I20260709 13:02:23.879949  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:23.984494  8655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.017s	sys 0.041s
W20260709 13:02:23.993109  8696 connection.cc:256] RPC call timeout handler was delayed by -1.2603756700000304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
I20260709 13:02:23.997673  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:23.941891 11291 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:23.884867 11281 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:23.873598 11171 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.617s	user 0.012s	sys 0.000s
I20260709 13:02:24.002535  8733 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:24.006261  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.027714  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:23.998843  8655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.017s	sys 0.041s
I20260709 13:02:23.970717 10950 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:23.921211 11274 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:23.895306 11068 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:23.899226 11247 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:24.006927  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.036s	sys 0.095s
W20260709 13:02:24.057969  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.036s	sys 0.095s
I20260709 13:02:24.033603  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:24.015491  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:24.008623 11189 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.019s	sys 0.049s
I20260709 13:02:24.056185  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:24.060336  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:24.065263  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49882us
I20260709 13:02:24.050606 10816 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 13:02:24.047830 10950 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:24.074839 11160 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:02:24.075845 11189 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.019s	sys 0.049s
I20260709 13:02:23.875520 11297 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.079754  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:24.085093  9327 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 717) took 2081 ms (client timeout 2004 ms). Trace:
W20260709 13:02:24.085779  9327 rpcz_store.cc:269] 0709 13:02:22.003738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.012048 (+  8310us) service_pool.cc:224] Handling call
0709 13:02:24.055976 (+2043928us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:24.084992 (+ 29016us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49629056}
I20260709 13:02:24.063647  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.087252  8657 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:24.085145  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.089213  8654 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:24.091107  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.028362 11236 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:24.092334  8650 raft_consensus.cc:2410] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:24.093660  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.095474  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:24.054746 11274 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:24.097872  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.012121 11230 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 13:02:24.048336  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 382) took 1636 ms. Trace:
I20260709 13:02:24.114662  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.096308  8652 raft_consensus.cc:2376] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 3. Re-sending same reply.
I20260709 13:02:24.028756 11280 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
W20260709 13:02:24.150010  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.100672  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:24.099125  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.126581  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.155411  8650 raft_consensus.cc:2410] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:24.128726  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:24.156288  8654 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:24.156455 11302 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 13:02:24.157792  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1687) took 1002 ms. Trace:
I20260709 13:02:24.158214  8578 rpcz_store.cc:276] 0709 13:02:23.155695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.155838 (+   143us) service_pool.cc:224] Handling call
0709 13:02:23.205772 (+ 49934us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.248008 (+ 42236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:23.248055 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:23.248684 (+   629us) log.cc:844] Serialized 26 byte log entry
0709 13:02:23.509013 (+260329us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:23.509077 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.509197 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:24.157572 (+648375us) raft_consensus.cc:1685] finished
0709 13:02:24.157591 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:24.157734 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11458,"raft.queue_time_us":63381,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"thread_start_us":195280,"threads_started":1,"wal-append.queue_time_us":700898}
I20260709 13:02:24.129894  8652 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:24.101162  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.134313  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1802) took 1145 ms. Trace:
I20260709 13:02:24.134512  8732 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:24.163065  8731 rpcz_store.cc:276] 0709 13:02:22.989305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.005390 (+ 16085us) service_pool.cc:224] Handling call
0709 13:02:24.134087 (+1128697us) spinlock_profiling.cc:243] Waited 94 ms on lock 0x7b540008a2d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:24.134242 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":216191232}
I20260709 13:02:24.163667  8652 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:02:24.136708  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1683) took 1960 ms. Trace:
I20260709 13:02:24.142879 11236 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:24.145257  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:24.164724  8651 raft_consensus.cc:2376] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 5. Re-sending same reply.
I20260709 13:02:24.164455  8579 rpcz_store.cc:276] 0709 13:02:22.176207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.269593 (+ 93386us) service_pool.cc:224] Handling call
0709 13:02:22.378917 (+109324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.028563 (+649646us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.136481 (+1107918us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54001694d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:24.136655 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1451754368}
I20260709 13:02:24.123612  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.166071  8656 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:24.130117 11285 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.887s	user 0.005s	sys 0.007s
I20260709 13:02:24.135977  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2299) took 2738 ms. Trace:
I20260709 13:02:24.168697  8659 rpcz_store.cc:276] 0709 13:02:21.397728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.434753 (+ 37025us) service_pool.cc:224] Handling call
0709 13:02:22.441837 (+1007084us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:22.442116 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.686478 (+1244362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.135766 (+449288us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000c80d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:24.135920 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2878175744}
I20260709 13:02:24.122614 11300 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.125571  8655 rpcz_store.cc:276] 0709 13:02:22.411770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.512470 (+100700us) service_pool.cc:224] Handling call
0709 13:02:22.512753 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.513007 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:22.782348 (+269341us) log.cc:844] Serialized 14 byte log entry
0709 13:02:24.047926 (+1265578us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:24.047989 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:24.048046 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.048129 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:24.048275 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64572,"prepare.queue_time_us":628685,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":94265,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":8086,"thread_start_us":1316864,"threads_started":3,"wal-append.queue_time_us":1127743}
I20260709 13:02:24.176067  8581 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:24.027493 11293 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
I20260709 13:02:24.182900  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:24.183240  8654 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:24.050311 11207 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:02:24.098966 11193 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:24.049705 11171 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.193282  8731 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:24.206491  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:24.207226  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.194783  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:24.196820 11207 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:24.198011  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.209625  8651 raft_consensus.cc:2410] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:24.211163  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:24.208050  8655 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:24.212445  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:24.215289  8731 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:24.216746  8577 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:24.216473  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:24.219832  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:24.199107 11271 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.487s	user 0.009s	sys 0.008s
I20260709 13:02:24.223217  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:24.221207  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:24.225623 11262 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:24.220942 11236 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.235127  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 522) took 2753 ms. Trace:
W20260709 13:02:24.232589 11140 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.739s	user 0.007s	sys 0.006s
I20260709 13:02:24.217855 11265 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:24.208765 11207 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:24.219643 11238 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.258438 11238 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:24.237283  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 511) took 1124 ms. Trace:
I20260709 13:02:24.259735  8575 rpcz_store.cc:276] 0709 13:02:23.112563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.118180 (+  5617us) service_pool.cc:224] Handling call
0709 13:02:23.118561 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.463532 (+344971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.235783 (+772251us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b540002e754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:24.237211 (+  1428us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":791077248}
I20260709 13:02:24.238670  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1796) took 2199 ms. Trace:
I20260709 13:02:24.262904  8726 rpcz_store.cc:276] 0709 13:02:22.039384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.040328 (+   944us) service_pool.cc:224] Handling call
0709 13:02:24.238427 (+2198099us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b5400229f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:24.238597 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1467339008}
W20260709 13:02:24.241945  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 526) took 2758 ms (client timeout 1947 ms). Trace:
W20260709 13:02:24.264379  8728 rpcz_store.cc:269] 0709 13:02:21.483929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.529990 (+ 46061us) service_pool.cc:224] Handling call
0709 13:02:24.241765 (+2711775us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540042fed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:24.241874 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234243840}
I20260709 13:02:24.265740  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:24.266508  8728 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:24.251888 11035 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:24.197537 11193 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.224642 11260 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:24.179697 11281 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:02:24.258718 11207 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.274886 11068 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:24.275209 11207 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:24.275781 11281 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:24.232029 10839 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:24.247520 11262 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:24.277581 11262 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:24.246480  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:24.248801 11148 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:24.283202 11171 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:24.283195 11148 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:24.283946 10839 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:24.284394  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.250281  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 379) took 1839 ms. Trace:
I20260709 13:02:24.252540  8730 rpcz_store.cc:276] 0709 13:02:21.481758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.482296 (+   538us) service_pool.cc:224] Handling call
0709 13:02:21.483646 (+  1350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.484074 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.484121 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:21.497762 (+ 13641us) log.cc:844] Serialized 26 byte log entry
0709 13:02:22.276099 (+778337us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:22.994812 (+718713us) log.cc:844] Serialized 14 byte log entry
0709 13:02:22.999687 (+  4875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.000156 (+   469us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.000265 (+   109us) raft_consensus.cc:1685] finished
0709 13:02:23.000281 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:24.234958 (+1234677us) spinlock_profiling.cc:243] Waited 986 us on lock 0x7b54000eae50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:24.235071 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12336,"prepare.queue_time_us":698886,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":5807,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":24243,"spinlock_wait_cycles":2266880,"thread_start_us":1331027,"threads_started":2,"wal-append.queue_time_us":870181,"wal-append.run_cpu_time_us":3695,"wal-append.run_wall_time_us":1627896}
I20260709 13:02:24.302254 11043 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:24.242853  8729 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:24.306244  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1789) took 2591 ms. Trace:
I20260709 13:02:24.306653  8729 rpcz_store.cc:276] 0709 13:02:21.715291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.717832 (+  2541us) service_pool.cc:224] Handling call
0709 13:02:24.242438 (+2524606us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:24.304927 (+ 62489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.306016 (+  1089us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b540042fed0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:24.306181 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5803907840}
I20260709 13:02:24.308359  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.256304 11140 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.228813 11271 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.309419  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:24.309988  8579 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:24.311131 11271 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:24.255676 11265 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.311846 11265 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:24.278337 11263 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:24.313226  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 544) took 1294 ms. Trace:
I20260709 13:02:24.276021 11264 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:24.313827 11264 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:24.285053  8578 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:24.315356  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:24.315953  8578 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:24.317173 11222 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:24.282696 11298 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.618s	user 0.010s	sys 0.004s
I20260709 13:02:24.287861 11261 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:24.287943 11260 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:24.288627  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:24.288904 11300 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:24.290154  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.349087  8581 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:24.350374  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.293051 11238 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:24.254753 11236 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:24.278719 11279 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.002s
W20260709 13:02:24.293606 11250 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.745s	user 0.008s	sys 0.005s
I20260709 13:02:24.294075 11082 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:24.294673 11193 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:24.296610 11274 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:24.298156 11218 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:24.300072  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.387991  8577 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:24.261595  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.311251 11140 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:24.284669 11148 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:24.312927  8580 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:24.313474 11263 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:24.286062 11281 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:24.321691 10938 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:24.309254  8576 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:24.305524  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.313611  8732 rpcz_store.cc:276] 0709 13:02:23.018646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.847421 (+828775us) service_pool.cc:224] Handling call
0709 13:02:24.134252 (+286831us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:24.163254 (+ 29002us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.313003 (+149749us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b540008a2d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:24.313169 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":656013056}
W20260709 13:02:24.278856  8725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.012s	sys 0.054s
I20260709 13:02:24.247609  8733 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:24.346191 11298 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:24.350884  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:24.351573  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:24.350860  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:24.353694  8579 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:24.367647  8578 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:24.367730  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:24.368587  8623 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.009s	sys 0.036s
I20260709 13:02:24.301862  8653 rpcz_store.cc:276] 0709 13:02:22.410667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.423865 (+ 13198us) service_pool.cc:224] Handling call
0709 13:02:24.249996 (+1826131us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b540018e7d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:24.250189 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2141916928}
W20260709 13:02:24.345646 11286 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.907s	user 0.019s	sys 0.023s
I20260709 13:02:24.370875  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:24.354460 11054 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.012s	sys 0.033s
W20260709 13:02:24.354326 11123 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.027s	sys 0.032s
I20260709 13:02:24.347350 11260 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:24.429118  8658 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.976s	user 0.008s	sys 0.079s
W20260709 13:02:24.374138 11234 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.019s	sys 0.031s
W20260709 13:02:24.466357 11156 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.002s	sys 0.001s
W20260709 13:02:24.466543 11276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.008s	sys 0.011s
I20260709 13:02:24.347813  8731 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:24.370711 11279 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:24.529107  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 199796us
W20260709 13:02:24.665275  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 564580us
I20260709 13:02:24.483273  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:24.504485  8624 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.052s	user 0.022s	sys 0.031s
W20260709 13:02:24.885833  8624 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.435s	user 0.028s	sys 0.040s
W20260709 13:02:24.723413  9297 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.010s	sys 0.011s
I20260709 13:02:24.539367  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 512) took 1382 ms. Trace:
W20260709 13:02:24.887910 11123 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.287s	user 0.032s	sys 0.038s
I20260709 13:02:24.888103  8580 rpcz_store.cc:276] 0709 13:02:23.157230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.157370 (+   140us) service_pool.cc:224] Handling call
0709 13:02:23.206742 (+ 49372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:24.391798 (+1185056us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.539002 (+147204us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54002cb0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:24.539289 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2542972800}
I20260709 13:02:24.884033  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:24.891835  8623 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.382s	user 0.013s	sys 0.039s
W20260709 13:02:24.892431  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 1.4351e+06us
I20260709 13:02:24.535228  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:24.893342  8584 raft_consensus.cc:1813] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Handling vote request from an unknown peer d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:24.724014  8724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.115s	user 0.012s	sys 0.008s
W20260709 13:02:24.893944  8724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.287s	user 0.014s	sys 0.010s
I20260709 13:02:24.536291  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:24.893748  8584 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:24.883184  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 532278us
W20260709 13:02:24.603101  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.009s	sys 0.007s
W20260709 13:02:24.898386  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.012s	sys 0.015s
W20260709 13:02:24.898878  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 850294us
W20260709 13:02:24.884635  8658 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.434s	user 0.009s	sys 0.079s
I20260709 13:02:24.885380  8655 raft_consensus.cc:2410] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 13:02:24.724798  9296 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.009s	sys 0.006s
I20260709 13:02:24.596336  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:24.476339  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1808) took 1022 ms. Trace:
I20260709 13:02:24.893704  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:24.539951  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:24.902177  8581 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:24.902828  8576 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:24.392549 11218 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:24.883997  8651 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:24.892490 11280 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:24.839121 11234 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.327s	user 0.027s	sys 0.033s
W20260709 13:02:24.897655  9297 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.137s	user 0.012s	sys 0.012s
W20260709 13:02:24.921118  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 1.17028e+06us
W20260709 13:02:24.884511 11286 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.474s	user 0.019s	sys 0.032s
W20260709 13:02:24.902304  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 306011us
I20260709 13:02:24.901069  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 383) took 2489 ms. Trace:
I20260709 13:02:24.901527  8579 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:24.729465  9325 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.041s	user 0.000s	sys 0.020s
I20260709 13:02:24.386914 11250 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:24.537611 11212 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
W20260709 13:02:24.840747 11276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.162s	user 0.010s	sys 0.014s
W20260709 13:02:24.901391  9296 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.011s	sys 0.007s
W20260709 13:02:24.741254 11303 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.392s	user 0.000s	sys 0.007s
W20260709 13:02:24.505456 11220 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.913s	user 0.008s	sys 0.044s
W20260709 13:02:24.475837  8725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.026s	user 0.012s	sys 0.054s
W20260709 13:02:24.901021  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:24.723630  9324 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.034s	user 0.009s	sys 0.011s
W20260709 13:02:24.735252 11095 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.003s	sys 0.016s
W20260709 13:02:24.724207 11283 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.013s	sys 0.011s
W20260709 13:02:24.839354 11156 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.150s	user 0.011s	sys 0.002s
W20260709 13:02:24.884012 11054 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.346s	user 0.019s	sys 0.036s
I20260709 13:02:24.902128  8733 rpcz_store.cc:276] 0709 13:02:23.454326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.857835 (+403509us) service_pool.cc:224] Handling call
0709 13:02:23.858172 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:24.475942 (+617770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.476253 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:24.917925 11035 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:24.729015 11160 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.535888 11263 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:24.948812 11304 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.628s	user 0.000s	sys 0.003s
I20260709 13:02:24.896181 11189 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:24.417419 11238 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:24.906306  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:24.740993 10950 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:25.147552  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 73882.9us
I20260709 13:02:25.173300  8731 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:24.896257 11275 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:25.266547  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 382500us
I20260709 13:02:24.946605  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:25.267202  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1819) took 1071 ms. Trace:
I20260709 13:02:25.268347  8658 rpcz_store.cc:276] 0709 13:02:22.412071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.514527 (+102456us) service_pool.cc:224] Handling call
0709 13:02:22.514898 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.515152 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:23.158564 (+643412us) log.cc:844] Serialized 14 byte log entry
0709 13:02:24.900714 (+1742150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:24.900767 (+    53us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:24.900810 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.900872 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:24.901015 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":225086,"prepare.queue_time_us":626829,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":103181,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":1876,"thread_start_us":2130090,"threads_started":3,"wal-append.queue_time_us":362156,"wal-append.run_cpu_time_us":1252,"wal-append.run_wall_time_us":1044522}
I20260709 13:02:25.269907  8584 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:25.270292  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:25.270429  9296 connection.cc:256] RPC call timeout handler was delayed by -1.4508077110000386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20260709 13:02:25.267891  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2276) took 1286 ms. Trace:
W20260709 13:02:25.277566  8727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.116s	user 0.003s	sys 0.044s
W20260709 13:02:25.331019  8727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.003s	sys 0.044s
W20260709 13:02:25.279449  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 378924us
W20260709 13:02:25.331555  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 437807us
W20260709 13:02:25.238186 11265 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.924s	user 0.012s	sys 0.000s
W20260709 13:02:25.233965 11313 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.330s	user 0.000s	sys 0.001s
I20260709 13:02:25.280118  8584 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:02:25.334409  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.366698  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1702) took 1213 ms. Trace:
I20260709 13:02:25.281733  8579 rpcz_store.cc:276] 0709 13:02:23.981692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.901027 (+919335us) service_pool.cc:224] Handling call
0709 13:02:24.901307 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:25.267633 (+366326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.267833 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:25.344482  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:25.367205  8584 rpcz_store.cc:276] 0709 13:02:24.153469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.279256 (+1125787us) service_pool.cc:224] Handling call
0709 13:02:25.279668 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:25.366352 (+ 86684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.366625 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:25.368010  9327 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
W20260709 13:02:25.264182 10839 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.000s	sys 0.043s
W20260709 13:02:25.233834 11252 consensus_meta.cc:287] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.428s	user 0.010s	sys 0.003s
W20260709 13:02:25.270968 11095 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.113s	user 0.008s	sys 0.029s
W20260709 13:02:25.380314  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 473990us
I20260709 13:02:25.353871 11263 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:25.371074 10839 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.087s	user 0.000s	sys 0.044s
I20260709 13:02:25.386407 11171 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:25.386653 10839 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:25.386960 11320 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 13:02:25.387253 11171 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:25.387646 10839 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:25.388505 11315 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 13:02:24.967687 10816 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.422s
I20260709 13:02:25.371024 11057 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 13:02:25.389729 11095 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:02:25.265774 11264 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.951s	user 0.005s	sys 0.006s
I20260709 13:02:25.361634  8650 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 13:02:25.271250 11283 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.506s	user 0.017s	sys 0.024s
I20260709 13:02:25.371016 10950 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:25.269930  9325 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.590s	user 0.012s	sys 0.031s
W20260709 13:02:24.967203 11207 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.691s	user 0.009s	sys 0.000s
W20260709 13:02:25.273614  9324 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.589s	user 0.013s	sys 0.025s
W20260709 13:02:25.280485  9296 connection.cc:256] RPC call timeout handler was delayed by -1.4130077110000343s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
I20260709 13:02:25.369930  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:25.227576 11236 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:24.910877 11300 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:25.372495  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 102264us
I20260709 13:02:25.375674  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:25.354496 11297 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.241s	user 0.043s	sys 0.036s
I20260709 13:02:25.376871  9328 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:25.377012  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1705) took 1183 ms. Trace:
I20260709 13:02:25.227627 11274 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:25.267268 11035 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:25.354640  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.955681 11140 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.423s
I20260709 13:02:25.276238 11321 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:25.395223  8698 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.860s	user 0.024s	sys 0.050s
W20260709 13:02:25.273916 11220 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.790s	user 0.022s	sys 0.057s
W20260709 13:02:25.396673  8580 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.030s	sys 0.040s
I20260709 13:02:25.365477 11123 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:25.374823 11156 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.331137 11160 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:25.389106 11171 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 13:02:25.278406 11238 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:25.365163 11189 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:25.373325 11265 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:25.174060 11260 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:25.203930 11218 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:25.375064 11324 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.145s
I20260709 13:02:25.398435  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:25.913481  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:25.913933  9330 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:25.914856  8698 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.379s	user 0.038s	sys 0.063s
I20260709 13:02:25.275243  8696 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:25.915314  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.37979e+06us
W20260709 13:02:25.915946  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 640687us
I20260709 13:02:25.397612  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:25.509260  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1684) took 2358 ms. Trace:
I20260709 13:02:25.274381  8731 rpcz_store.cc:276] 0709 13:02:24.195729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.288281 (+ 92552us) service_pool.cc:224] Handling call
0709 13:02:25.267140 (+978859us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:25.917511  8698 connection.cc:256] RPC call timeout handler was delayed by -1.0720195940000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1743s for a response
I20260709 13:02:25.910943  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1810) took 2439 ms. Trace:
W20260709 13:02:25.446338  8729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.013s	sys 0.037s
W20260709 13:02:25.390064 11322 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
W20260709 13:02:25.394411 10938 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.911s	user 0.009s	sys 0.002s
W20260709 13:02:25.919323  9296 connection.cc:256] RPC call timeout handler was delayed by -1.032307711000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25s for a response
I20260709 13:02:25.383561 11298 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:25.390151 11095 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:26.000142  8698 connection.cc:256] RPC call timeout handler was delayed by -1.0126195940000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
I20260709 13:02:25.365206 11323 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:25.380426 11054 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:25.957603  8579 rpcz_store.cc:276] 0709 13:02:24.193799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.367572 (+1173773us) service_pool.cc:224] Handling call
0709 13:02:25.376239 (+  8667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:25.376507 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:25.376552 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:25.376616 (+    64us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:25.376674 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.376754 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.376931 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:25.919581  8582 rpcz_store.cc:276] 0709 13:02:23.151568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.151819 (+   251us) service_pool.cc:224] Handling call
0709 13:02:23.152264 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.265679 (+113415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:23.265726 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:23.278495 (+ 12769us) log.cc:844] Serialized 26 byte log entry
0709 13:02:23.500501 (+222006us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:23.500561 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.500688 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:25.461508 (+1960820us) raft_consensus.cc:1685] finished
0709 13:02:25.508993 (+ 47485us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.509196 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":87127,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"spinlock_wait_cycles":15104,"thread_start_us":277611,"threads_started":2,"wal-append.queue_time_us":1000599}
I20260709 13:02:25.962679  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2167) took 1872 ms. Trace:
W20260709 13:02:25.895076 11316 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.668s	user 0.000s	sys 0.001s
W20260709 13:02:25.912616 11317 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.319s	user 0.000s	sys 0.000s
W20260709 13:02:26.001685  8729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.313s	user 0.024s	sys 0.051s
W20260709 13:02:25.911942 11312 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.361s	user 0.000s	sys 0.000s
W20260709 13:02:25.994302  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 408132us
I20260709 13:02:26.011386  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:25.395848 11247 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.160s	user 0.032s	sys 0.046s
W20260709 13:02:25.815559 11334 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.575s	user 0.000s	sys 0.001s
W20260709 13:02:25.918666 11318 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.659s	user 0.000s	sys 0.001s
I20260709 13:02:25.776129 11208 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:25.997035 11249 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:25.710940 11306 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.668128 11275 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.008322 11250 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.048864  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:26.033332 11247 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.798s	user 0.042s	sys 0.060s
W20260709 13:02:25.880259 11311 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.490s	user 0.000s	sys 0.001s
W20260709 13:02:25.391100 11262 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.113s	user 0.010s	sys 0.003s
I20260709 13:02:26.030905  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:25.918241 11274 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:25.918746 11276 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:25.587779 11310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.098s	user 0.026s	sys 0.046s
I20260709 13:02:25.509158 11279 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:25.942152 11283 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:26.008845 11035 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:26.000694 11123 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.383280 11252 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:25.999378 11297 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.886s	user 0.062s	sys 0.076s
I20260709 13:02:26.029114 11218 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:26.029635 11260 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:26.027976 11307 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.397223 11264 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.032189 11323 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:26.057710 11306 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.064189 11276 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:26.058844 11310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.619s	user 0.027s	sys 0.047s
I20260709 13:02:26.025278 11300 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.020584 11321 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.061059 11275 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:26.099900 11338 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 13:02:26.103123 11323 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:26.138172  8548 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.030s	sys 0.058s
W20260709 13:02:25.987684  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.140081  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.140370  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.141175  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.141098  8578 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:26.142411  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.000495  8580 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.110s	user 0.047s	sys 0.067s
I20260709 13:02:26.143436  8583 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 5 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:26.009763  8650 rpcz_store.cc:276] 0709 13:02:24.089872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.177198 (+ 87326us) service_pool.cc:224] Handling call
0709 13:02:24.177575 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:25.395161 (+1217586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.962292 (+567131us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540007dad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:25.962588 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2721819008}
W20260709 13:02:26.145715  8576 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.000s	sys 0.009s
W20260709 13:02:26.146313  8576 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.000s	sys 0.009s
I20260709 13:02:26.148233  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 528) took 4660 ms. Trace:
I20260709 13:02:25.993249  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:26.149680  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.005s	sys 0.008s
W20260709 13:02:26.149787  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 156524us
I20260709 13:02:26.150581  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1813) took 1966 ms. Trace:
W20260709 13:02:26.150274  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.005s	sys 0.009s
W20260709 13:02:25.549803  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.151769  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 768168us
I20260709 13:02:26.151189  8730 rpcz_store.cc:276] 0709 13:02:24.183982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.302634 (+118652us) service_pool.cc:224] Handling call
0709 13:02:24.303031 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:24.303435 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:24.303469 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:24.303987 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 13:02:24.829596 (+525609us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:24.829657 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.829759 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.149967 (+1320208us) raft_consensus.cc:1685] finished
0709 13:02:26.150009 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.150343 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5550,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"thread_start_us":163063,"threads_started":1,"wal-append.queue_time_us":895476}
I20260709 13:02:26.154917  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:26.156561  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.157763  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.824s	user 0.008s	sys 0.026s
I20260709 13:02:26.001159  8727 rpcz_store.cc:276] 0709 13:02:23.471222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.052091 (+580869us) service_pool.cc:224] Handling call
0709 13:02:24.052478 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:24.052915 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:24.052957 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:24.061679 (+  8722us) log.cc:844] Serialized 26 byte log entry
0709 13:02:25.337256 (+1275577us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:25.337319 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.337442 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:25.910645 (+573203us) raft_consensus.cc:1685] finished
0709 13:02:25.910673 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.910853 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2797,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":24100,"thread_start_us":1175801,"threads_started":1,"wal-append.queue_time_us":1808826}
I20260709 13:02:26.158218  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:26.159642  8653 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
W20260709 13:02:26.158255  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.008s	sys 0.026s
I20260709 13:02:26.161105  8651 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 13:02:26.161367  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 827811us
I20260709 13:02:26.163404  8650 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:26.148857  8724 rpcz_store.cc:276] 0709 13:02:21.487658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.613788 (+126130us) service_pool.cc:224] Handling call
0709 13:02:21.629922 (+ 16134us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.660021 (+ 30099us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:21.660055 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:23.563476 (+1903421us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b2c000b2cd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:02:23.563840 (+   364us) log.cc:844] Serialized 26 byte log entry
0709 13:02:24.894432 (+1330592us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:24.894486 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.894564 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.147123 (+1252559us) raft_consensus.cc:1685] finished
0709 13:02:26.147193 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.147880 (+   687us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23453,"raft.queue_time_us":30133,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":33505,"spinlock_wait_cycles":6093568,"thread_start_us":1286918,"threads_started":1,"wal-append.queue_time_us":1457368}
I20260709 13:02:25.509521  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2122) took 3187 ms. Trace:
W20260709 13:02:26.139390  8548 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.030s	sys 0.058s
W20260709 13:02:26.165303  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 896740us
I20260709 13:02:26.165988  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:26.166193  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:26.151279  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:26.170050  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 729) took 2714 ms. Trace:
I20260709 13:02:26.170742  9325 rpcz_store.cc:276] 0709 13:02:23.455562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.459790 (+  4228us) service_pool.cc:224] Handling call
0709 13:02:23.496168 (+ 36378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.549842 (+ 53674us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:23.554814 (+  4972us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:23.608697 (+ 53883us) log.cc:844] Serialized 26 byte log entry
0709 13:02:25.971223 (+2362526us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:25.974372 (+  3149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.997748 (+ 23376us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.169313 (+171565us) raft_consensus.cc:1685] finished
0709 13:02:26.169387 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.169826 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3168,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":19167,"thread_start_us":2243349,"threads_started":1,"wal-append.queue_time_us":1745185}
I20260709 13:02:26.055608  9328 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:26.174752  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.038456  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.176172  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.176546  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:26.177197  9328 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:26.177771  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.164654  8725 rpcz_store.cc:276] 0709 13:02:22.273187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.302437 (+1029250us) service_pool.cc:224] Handling call
0709 13:02:23.340776 (+ 38339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.410657 (+ 69881us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:23.450040 (+ 39383us) log.cc:844] Serialized 14 byte log entry
0709 13:02:25.375394 (+1925354us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:25.383026 (+  7632us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:25.390614 (+  7588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.397013 (+  6399us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.460894 (+ 63881us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82211,"prepare.queue_time_us":30982,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":99014,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":18488,"thread_start_us":1814864,"threads_started":1,"wal-append.queue_time_us":982523,"wal-append.run_cpu_time_us":1473,"wal-append.run_wall_time_us":1018369}
I20260709 13:02:26.178819  9331 raft_consensus.cc:2376] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 3. Re-sending same reply.
I20260709 13:02:26.179847  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:26.180316  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.180919  9325 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 2: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:26.181789  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.040174 10938 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:26.166152  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 724) took 3133 ms. Trace:
W20260709 13:02:26.054581 10816 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.015s	sys 0.014s
I20260709 13:02:26.020313 11156 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.073562 11160 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.252920 11160 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:26.253845 11160 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.254700 11156 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.039142 11207 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 13:02:26.256896  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.258129 11057 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.258448  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 518) took 2397 ms. Trace:
I20260709 13:02:26.258996 11057 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:26.260070  8550 leader_election.cc:336] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.089s (SENT)
I20260709 13:02:26.044266 11208 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:26.201920 11350 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:26.168617  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:26.231424  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1689) took 2084 ms. Trace:
I20260709 13:02:26.240845 11353 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:26.229020  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:26.239174  9328 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:26.244244 11148 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.008s	sys 0.056s
I20260709 13:02:26.244165 11094 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:26.248724 10816 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.015s	sys 0.015s
I20260709 13:02:26.249275 10938 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:26.250959 11238 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.246502  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1696) took 2093 ms. Trace:
I20260709 13:02:26.260874  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:26.259006  8583 rpcz_store.cc:276] 0709 13:02:23.861478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.861698 (+   220us) service_pool.cc:224] Handling call
0709 13:02:23.873996 (+ 12298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.144085 (+2270089us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.258129 (+114044us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b54001680d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:26.258371 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5216864512}
I20260709 13:02:26.063102 11262 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.078452 11123 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:26.274466  8580 rpcz_store.cc:276] 0709 13:02:24.146757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.889957 (+743200us) service_pool.cc:224] Handling call
0709 13:02:24.890347 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:24.890589 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:26.143566 (+1252977us) log.cc:844] Serialized 14 byte log entry
0709 13:02:26.230311 (+ 86745us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:26.230421 (+   110us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:26.230588 (+   167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.230771 (+   183us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.231047 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1204604,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":567,"thread_start_us":1338138,"threads_started":2,"wal-append.queue_time_us":86561}
I20260709 13:02:26.316368 11358 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:26.326377 11359 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:26.327035 11359 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.327472 11359 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:26.333312  8548 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.641s (SENT)
I20260709 13:02:26.314925  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.334003  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:26.334908 11057 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.335484  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:26.298733  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:26.294054 11148 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.880s	user 0.009s	sys 0.056s
I20260709 13:02:26.337076  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:26.314086  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 163053us
I20260709 13:02:26.287218 11353 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:26.342979  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:26.297466  8576 rpcz_store.cc:276] 0709 13:02:24.151774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.264172 (+1112398us) service_pool.cc:224] Handling call
0709 13:02:25.330458 (+ 66286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.147053 (+816595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:26.147129 (+    76us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:26.148241 (+  1112us) log.cc:844] Serialized 26 byte log entry
0709 13:02:26.233820 (+ 85579us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:26.233876 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.234011 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.236389 (+  2378us) raft_consensus.cc:1685] finished
0709 13:02:26.236414 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.244800 (+  8386us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24313,"raft.queue_time_us":49540,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":7838,"thread_start_us":876505,"threads_started":2,"wal-append.queue_time_us":85538}
W20260709 13:02:26.354146  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.348883 11094 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.361944 11057 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:02:26.364550 11263 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.978s	user 0.013s	sys 0.000s
W20260709 13:02:26.365535  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 345833us
I20260709 13:02:26.365633 11265 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.061733 11095 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.369769  8730 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:26.369992  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:26.371302  8654 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:26.371435 11095 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:26.373865  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 207961us
I20260709 13:02:26.373574 11350 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.376768  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:26.237845 11356 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.121s
W20260709 13:02:26.364463  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 104499us
I20260709 13:02:26.392463 11222 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.383837  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2273) took 2470 ms. Trace:
I20260709 13:02:26.396106  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:26.396437 11321 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.397567 11252 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.398653  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:26.399443 11306 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.399734 11238 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.400074 11300 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.400701 11306 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:26.348223 11353 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:26.405939 11262 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.056404 11249 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:26.421219  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 87981.7us
I20260709 13:02:26.190220  9324 rpcz_store.cc:276] 0709 13:02:23.032665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.039356 (+  6691us) service_pool.cc:224] Handling call
0709 13:02:23.039733 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.522053 (+482320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:23.534158 (+ 12105us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:23.615029 (+ 80871us) log.cc:844] Serialized 26 byte log entry
0709 13:02:25.971112 (+2356083us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:25.974312 (+  3200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.997706 (+ 23394us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.165481 (+167775us) raft_consensus.cc:1685] finished
0709 13:02:26.165523 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.165883 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16895,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":2709172,"threads_started":2,"wal-append.queue_time_us":1583400}
I20260709 13:02:26.381934  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:26.410373  8658 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:26.412214 11220 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.412918 11250 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:26.416774  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1814) took 2232 ms. Trace:
I20260709 13:02:26.382583  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:26.382633 11133 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:26.397661 11281 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.362202  8651 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:26.400555 11238 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.423897 11357 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:26.090226 11307 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.425370 11264 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.425886  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:26.426795  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:26.427361  8659 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:26.377938  8731 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:26.424367 11054 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:26.431030 11276 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.434633 11250 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.379451  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:26.411520  8576 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:26.382349  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:26.394410  8578 rpcz_store.cc:276] 0709 13:02:23.913800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.900459 (+986659us) service_pool.cc:224] Handling call
0709 13:02:24.900787 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.142359 (+1241572us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.383471 (+241112us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54003e7454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:26.383721 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2850280320}
I20260709 13:02:26.382578  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:26.364491 11094 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:26.377364 11350 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.421303 11306 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.468149  8657 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:26.469449  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:26.460893 11171 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:26.473747  9327 raft_consensus.cc:2376] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5. Re-sending same reply.
I20260709 13:02:26.474247 11171 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.474727 11171 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:26.442179 11263 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:26.425439 11275 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.089963 11274 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.436038 11133 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:26.443778  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 281109us
W20260709 13:02:26.446511  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 69969us
W20260709 13:02:26.447479  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 278727us
I20260709 13:02:26.428687  8729 rpcz_store.cc:276] 0709 13:02:24.184543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.307516 (+122973us) service_pool.cc:224] Handling call
0709 13:02:24.307812 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:24.308039 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:24.688597 (+380558us) log.cc:844] Serialized 14 byte log entry
0709 13:02:26.358249 (+1669652us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:26.415870 (+ 57621us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:26.416054 (+   184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.416204 (+   150us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.416482 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12595,"prepare.queue_time_us":937234,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":362838,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":150,"thread_start_us":1956135,"threads_started":3,"wal-append.queue_time_us":1091566}
I20260709 13:02:26.434078 11356 consensus_queue.cc:1243] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:26.412364 10938 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.474947  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:26.746683  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.626384  9328 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
W20260709 13:02:26.837443  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 366073us
W20260709 13:02:26.743096  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.845088  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.500581  8548 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.113s (SENT)
I20260709 13:02:26.734760  8659 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
W20260709 13:02:26.844240  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 392641us
I20260709 13:02:26.476987  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:26.605330  9331 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:26.479094  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:26.847064  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.847986  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.484830  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.851579  8657 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:26.489905  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:26.498503  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:26.901861  8654 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.530s	user 0.009s	sys 0.005s
W20260709 13:02:26.490394  9330 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.540s	user 0.006s	sys 0.009s
I20260709 13:02:26.452719  9325 raft_consensus.cc:2410] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:02:26.485154  8727 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:26.913285  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:26.914068  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.487953  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:26.900590  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:26.809350 11260 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.721s	user 0.008s	sys 0.002s
I20260709 13:02:26.917913  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:26.919888  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:26.921239 11366 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:26.922891 11249 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:26.923321 11250 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.923758 11249 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:26.924304 11249 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:26.924595 11307 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.924778 11274 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:26.925592 11264 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.926367 11264 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.928397 11342 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:26.929380 11342 consensus_queue.cc:1243] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:26.929641  8651 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:26.632731 11160 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:26.930161 11342 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:02:26.932561  8479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 306) took 15288 ms (client timeout 15000 ms). Trace:
I20260709 13:02:26.932813 11264 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.934422 11367 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.934916 10938 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:26.935480 11140 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:26.935168 11367 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.936178 11140 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:26.493580  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.040s	sys 0.035s
I20260709 13:02:26.817713 11324 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:26.930176  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 429533us
I20260709 13:02:26.939204  8654 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:26.506035 11306 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:26.834908 11238 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:26.958138  8728 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:26.963923 11082 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:26.964519 11082 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.964941 11082 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:26.491297 11054 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.965776  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:26.966612  8733 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:26.969569 11298 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.929096  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:26.970326 11298 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.971426 11357 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:26.972584  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:26.924019 11250 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:26.973737 11250 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:26.974162 11250 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:26.974699  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:26.978114  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58390.9us
I20260709 13:02:26.941606 11189 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:26.944188  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:26.944783 11247 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.929570 11279 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:26.934453  8482 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:27.053748  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:26.948297 11095 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.576s	user 0.009s	sys 0.005s
I20260709 13:02:26.949108  8726 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:26.949747 11035 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.704s	user 0.012s	sys 0.001s
I20260709 13:02:26.479377 11344 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.480597  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:27.068552 11171 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.068912  8578 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:27.069437  9324 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:26.475088 11222 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:26.956391 11249 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:26.933125  8479 rpcz_store.cc:269] 0709 13:02:11.643538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.643762 (+   224us) service_pool.cc:224] Handling call
0709 13:02:26.932284 (+15288522us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:20.680051 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:20.680735 (+   684us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:20.680834 (+    99us) write_op.cc:435] Acquired schema lock
0709 13:02:20.680869 (+    35us) tablet.cc:662] Decoding operations
0709 13:02:20.681365 (+   496us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:20.716296 (+ 34931us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:20.730569 (+ 14273us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:02:20.805041 (+ 74472us) tablet.cc:701] Row locks acquired
0709 13:02:20.805073 (+    32us) write_op.cc:260] PREPARE: finished
0709 13:02:20.805288 (+   215us) write_op.cc:270] Start()
0709 13:02:20.805470 (+   182us) write_op.cc:276] Timestamp: P: 1783602140805228 usec, L: 0
0709 13:02:20.805508 (+    38us) op_driver.cc:348] REPLICATION: starting
0709 13:02:21.104184 (+298676us) log.cc:844] Serialized 912 byte log entry
0709 13:02:23.334330 (+2230146us) op_driver.cc:464] REPLICATION: finished
0709 13:02:24.438467 (+1104137us) write_op.cc:301] APPLY: starting
0709 13:02:24.438586 (+   119us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:24.438740 (+   154us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:24.438896 (+   156us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:24.439404 (+   508us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:24.439671 (+   267us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 13:02:24.439733 (+    62us) write_op.cc:312] APPLY: finished
0709 13:02:24.440017 (+   284us) log.cc:844] Serialized 29 byte log entry
0709 13:02:26.096283 (+1656266us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:26.098256 (+  1973us) write_op.cc:454] Released schema lock
0709 13:02:26.098879 (+   623us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:26.372565 (+273686us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:26.372644 (+    79us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":964008,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1094048,"apply.run_cpu_time_us":75872,"apply.run_wall_time_us":1934899,"num_ops":2,"prepare.queue_time_us":543274,"prepare.run_cpu_time_us":12521,"prepare.run_wall_time_us":720638,"raft.queue_time_us":583001,"raft.run_cpu_time_us":69434,"raft.run_wall_time_us":2085887,"replication_time_us":2491685,"thread_start_us":4572728,"threads_started":5,"wal-append.queue_time_us":2167950,"wal-append.run_cpu_time_us":162168,"wal-append.run_wall_time_us":2474315}]]}
W20260709 13:02:26.980782  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51761.2us
I20260709 13:02:26.940645 11324 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:26.990890  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 738) took 1316 ms. Trace:
I20260709 13:02:27.073825  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 712) took 1042 ms. Trace:
I20260709 13:02:26.806149 11359 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.798022 11123 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.501727 11281 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.052541  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:26.741050 10950 consensus_meta.cc:287] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.721s	user 0.008s	sys 0.005s
I20260709 13:02:26.493517 11309 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:26.737854 10816 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:26.946733 11133 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:26.480098  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:26.464457 11262 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.490774 11358 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:27.051595  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:27.056943  8650 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 13:02:26.952137  8684 heartbeater.cc:646] Failed to heartbeat to 127.7.78.126:44983 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.7.78.126:44983 timed out after 15.000s (SENT)
I20260709 13:02:26.953315  8732 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:02:27.060413 11279 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.060776  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.061522 11247 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.061359 11035 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.061256  8482 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:27.063019  8575 raft_consensus.cc:2410] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:02:26.479914 11212 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.586s
I20260709 13:02:27.065440  8733 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:27.065637  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 92461.3us
I20260709 13:02:26.485458 11283 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:26.500185 11353 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.066465  9327 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:27.067075  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:26.480732 11345 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.595s
I20260709 13:02:26.707139  8725 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:27.074306  9326 rpcz_store.cc:276] 0709 13:02:26.012035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.048467 (+ 36432us) service_pool.cc:224] Handling call
0709 13:02:26.984961 (+936494us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b5400867754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:27.054368 (+ 69407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2041603328}
I20260709 13:02:26.476343 11364 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.452103 11220 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:27.058143  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 1.14109e+06us
I20260709 13:02:27.215740 10938 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:27.216269  8729 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:27.224640  8726 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:27.216982  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 715) took 1183 ms. Trace:
I20260709 13:02:27.218171  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.218508  8655 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:27.067325  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.231002  9327 rpcz_store.cc:276] 0709 13:02:26.034880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.065579 (+1030699us) service_pool.cc:224] Handling call
0709 13:02:27.066087 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:27.216641 (+150554us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.216890 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.220625 11353 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.221729  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:27.168951  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68341us
I20260709 13:02:27.222831  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 737) took 1825 ms. Trace:
I20260709 13:02:27.222457 11283 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.082509  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:27.170871  8728 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:27.076315  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.075212  9331 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:27.217552 11260 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:27.234934  9330 rpcz_store.cc:276] 0709 13:02:25.397937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.398119 (+   182us) service_pool.cc:224] Handling call
0709 13:02:27.222734 (+1824615us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.239019  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 711) took 1227 ms. Trace:
I20260709 13:02:27.214811  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:27.239818  9331 rpcz_store.cc:276] 0709 13:02:26.011833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.074631 (+1062798us) service_pool.cc:224] Handling call
0709 13:02:27.075005 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.238720 (+163715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.238948 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.244079  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.243098  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.245688  8577 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:27.223330 11262 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.225490 11309 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.261656 11309 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:27.235833 11140 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.261300  8576 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 13:02:27.215140 11171 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.241578 11300 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.216982  9323 raft_consensus.cc:2376] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 3. Re-sending same reply.
I20260709 13:02:27.230403 11367 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.256098  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.256914  9328 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:27.303807  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 718) took 1146 ms. Trace:
I20260709 13:02:27.254264  8727 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:27.245566 11095 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:27.257589  8656 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:27.258224 11306 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.275158  9325 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:27.303411  8724 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:27.258888 11133 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.259770 11332 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.260221 11364 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } attrs { replace: true } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.301628 11353 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:27.252522  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.230975 10816 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:27.276396  9329 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:27.299065  8728 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:27.299511  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:27.075632 11218 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.193s
I20260709 13:02:27.247221  8730 raft_consensus.cc:2376] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 4. Re-sending same reply.
I20260709 13:02:27.268959  8482 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:27.267370 11298 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.305382  8733 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:27.313266  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2305) took 1151 ms. Trace:
I20260709 13:02:27.257663 11260 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.260793  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.326260  8726 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:27.319031  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:27.326967  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:27.327405  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.324631  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.088570 11344 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.327405  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 719) took 1169 ms. Trace:
I20260709 13:02:27.304186  9328 rpcz_store.cc:276] 0709 13:02:26.157811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.902438 (+744627us) service_pool.cc:224] Handling call
0709 13:02:27.303727 (+401289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.323350  8650 rpcz_store.cc:276] 0709 13:02:26.174680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.176113 (+  1433us) service_pool.cc:224] Handling call
0709 13:02:27.307468 (+1131355us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54001310d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:27.307617 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2021906688}
I20260709 13:02:27.074417 11324 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.332237  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:27.332883  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.328891  9329 rpcz_store.cc:276] 0709 13:02:26.158235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.071803 (+913568us) service_pool.cc:224] Handling call
0709 13:02:27.235914 (+164111us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.327151 (+ 91237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.327347 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.082062 11208 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.393471  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:27.304992  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59178.1us
I20260709 13:02:27.311448 11160 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:27.402473  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 350220us
I20260709 13:02:27.314738 11171 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.137s
I20260709 13:02:27.307229  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 714) took 1293 ms. Trace:
I20260709 13:02:27.392781  8482 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 3, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:27.302783 11300 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:27.459010  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.330807  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 710) took 1319 ms. Trace:
I20260709 13:02:27.476684  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:27.463176  8729 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:27.481022  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.486568  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.421526  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:27.501597  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 258606us
I20260709 13:02:27.499336 11344 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:27.402940 11140 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:27.512985  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:27.514307  8651 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:27.511888  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:27.515779  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:27.542320  8652 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:27.507483  8482 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 4 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:27.543538 11281 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.543795  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:27.320451 11332 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.314096 11082 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.543454  8482 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:27.550851  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:27.543905  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:27.544530  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:27.558634  8659 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:27.549779  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.477965  9325 rpcz_store.cc:276] 0709 13:02:26.011593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.950393 (+938800us) service_pool.cc:224] Handling call
0709 13:02:27.059380 (+108987us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.314455 (+255075us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.326825 (+ 12370us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.073961  9322 rpcz_store.cc:276] 0709 13:02:25.539803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.550375 (+ 10572us) service_pool.cc:224] Handling call
0709 13:02:25.798755 (+248380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.334565 (+535810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:26.334634 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:26.335450 (+   816us) log.cc:844] Serialized 26 byte log entry
0709 13:02:26.834772 (+499322us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:26.834824 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.834925 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.855366 (+ 20441us) raft_consensus.cc:1685] finished
0709 13:02:26.855398 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.855637 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12552,"raft.queue_time_us":329829,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":102654,"thread_start_us":478123,"threads_started":2,"wal-append.queue_time_us":517916}
I20260709 13:02:27.566769  8657 raft_consensus.cc:2376] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3. Re-sending same reply.
I20260709 13:02:27.564114  8482 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 5 to 6, term changed from 3 to 5, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68), VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New cstate: current_term: 5 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:27.516528  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 54761.5us
W20260709 13:02:27.484269 11357 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.512s	user 0.010s	sys 0.006s
I20260709 13:02:27.420691 11095 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:27.585206  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.586159 11123 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.586833  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:27.586336  8482 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:27.587301  8725 raft_consensus.cc:2410] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 13:02:27.509444 11274 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.584s	user 0.012s	sys 0.004s
I20260709 13:02:27.580591  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:27.586051  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:27.591051  8651 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:27.569479 11250 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.595s	user 0.006s	sys 0.007s
I20260709 13:02:27.549582 11324 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.461158  9323 rpcz_store.cc:276] 0709 13:02:26.012343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.405241 (+392898us) service_pool.cc:224] Handling call
0709 13:02:27.305783 (+900542us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.546408 11035 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.594780  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:27.595168  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:27.550314 11359 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.516662 11377 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.573849 11054 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 13:02:27.590674 11220 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.545527 10816 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.603827  8579 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:27.604018  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:27.605099 11306 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.606266 11260 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:27.656157  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.607829 11247 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.608373  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.609422 11082 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.610471  8732 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:02:27.610801  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110184us
I20260709 13:02:27.611230 11250 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.574903 11353 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:27.617236  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:27.615854  8729 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:27.638729  8657 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:27.639963  8654 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 2.
I20260709 13:02:27.641568  8733 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:27.500386 11160 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.677475 11212 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260709 13:02:27.678584 11354 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 13:02:27.646948  8482 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:27.602042 11357 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:27.637374 10938 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:27.610515 11262 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:27.664472  8548 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.542s (SENT)
I20260709 13:02:27.658375 11220 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.689091  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:27.691256  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.691259  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:27.694363  8652 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:27.669425 10816 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.700476  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.683964 11140 raft_consensus.cc:2764] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:02:27.707872 11095 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:27.708395 11375 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:02:27.715395  8653 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:27.696352  8482 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 3 to 5, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 5 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:27.720965  8482 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:27.729710  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 163929us
I20260709 13:02:27.690203 11279 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.659972 11377 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:27.733492  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 69103.5us
I20260709 13:02:27.734557 11250 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:27.738092  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 238998us
I20260709 13:02:27.681267 11035 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 13:02:27.735666  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.713014 11353 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:27.750403  8726 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:27.706348 11262 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:27.763543 11379 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:27.755158 11357 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:27.756860  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.698596 11220 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.764205 11379 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:27.762638 11279 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.767138 11279 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.767712 11379 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:27.769565  8581 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:27.769791 11279 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:27.770035 11374 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 13:02:27.771060  8566 tablet_service.cc:1558] Processing DeleteTablet for tablet 2947646ebfc34bd7a5a55312f5812ee2 with delete_type TABLET_DATA_TOMBSTONED (TS a0d703ee18464475a290425976246555 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41978
W20260709 13:02:27.772495  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 98316.2us
I20260709 13:02:27.776491  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.763795  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:27.763880 11383 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:27.777120  8726 raft_consensus.cc:1813] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Handling vote request from an unknown peer a0d703ee18464475a290425976246555
I20260709 13:02:27.753281 11307 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:27.822332  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2313) took 1443 ms. Trace:
I20260709 13:02:27.869478  8726 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:27.778272  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 89571.2us
I20260709 13:02:27.797683  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2307) took 1622 ms. Trace:
W20260709 13:02:27.793066  9297 leader_election.cc:336] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.689s (SENT)
W20260709 13:02:27.786033  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1704) took 3632 ms (client timeout 4760 ms). Trace:
I20260709 13:02:27.777172 11300 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:27.800634 11279 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20260709 13:02:27.793951 11276 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
I20260709 13:02:27.894922 11140 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:27.895840 11364 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.870118  8579 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:27.897876 11344 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:27.922264  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2145) took 1432 ms. Trace:
W20260709 13:02:27.897833 11368 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.622s	user 0.000s	sys 0.004s
W20260709 13:02:27.790196 11380 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260709 13:02:27.902132  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 213114us
I20260709 13:02:27.914317  8656 rpcz_store.cc:276] 0709 13:02:26.378956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.382757 (+  3801us) service_pool.cc:224] Handling call
0709 13:02:26.383174 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.312438 (+929264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.821990 (+509552us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b54000596d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:27.822246 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1985668480}
I20260709 13:02:27.809365  8729 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:27.922712  8726 rpcz_store.cc:276] 0709 13:02:26.489390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.776199 (+1286809us) service_pool.cc:224] Handling call
0709 13:02:27.922187 (+145988us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.903605  9297 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:27.876075  8732 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
W20260709 13:02:27.905230  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 292392us
W20260709 13:02:27.831724  8727 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.514s	user 0.009s	sys 0.000s
I20260709 13:02:27.784274  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:27.873034  8548 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.630s (SENT)
W20260709 13:02:27.895668  8575 rpcz_store.cc:269] 0709 13:02:24.154036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.483008 (+328972us) service_pool.cc:224] Handling call
0709 13:02:25.729685 (+1246677us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b54002cb428. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9f3924 00007f140ebdc227 00007f140ebdbab8 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6
0709 13:02:27.785804 (+2056119us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54002cb5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:27.785957 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3049372416}
W20260709 13:02:27.807335 11309 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.545s	user 0.008s	sys 0.003s
I20260709 13:02:27.784827  8725 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:27.904429  8655 rpcz_store.cc:276] 0709 13:02:26.175752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.177520 (+  1768us) service_pool.cc:224] Handling call
0709 13:02:26.376497 (+198977us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:26.387454 (+ 10957us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.232509 (+845055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.797368 (+564859us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:27.797596 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1913664128}
I20260709 13:02:27.935483  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.931332  8727 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:27.936578  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2148) took 1177 ms. Trace:
W20260709 13:02:27.928891  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.930487  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2150) took 1170 ms. Trace:
I20260709 13:02:27.938486  8729 rpcz_store.cc:276] 0709 13:02:26.760170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.718986 (+958816us) service_pool.cc:224] Handling call
0709 13:02:27.930390 (+211404us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:27.893500  8697 connection.cc:256] RPC call timeout handler was delayed by -3.4679057090000356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
I20260709 13:02:27.779350 11035 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:27.808899 11377 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.936916  8727 rpcz_store.cc:276] 0709 13:02:26.759064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.995477 (+236413us) service_pool.cc:224] Handling call
0709 13:02:27.936512 (+941035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.873451 11383 consensus_queue.cc:1243] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 13:02:27.944639  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 160543us
I20260709 13:02:27.915799 11307 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:27.795503 11332 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:27.784535 11283 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.156s
W20260709 13:02:27.948760  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 75756us
I20260709 13:02:27.946801  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:27.953310  8697 leader_election.cc:336] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.518s (SENT)
W20260709 13:02:27.957439  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.947022  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:27.940627 11358 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.967803  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:27.941586 11035 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:27.972616  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:27.972846  8697 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:27.970415  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:27.946252 11189 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:27.977193  8732 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:27.797974 11353 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:27.982020  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:27.979763  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 185507us
W20260709 13:02:27.988641  9296 connection.cc:256] RPC call timeout handler was delayed by -2.2979973169999539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20260709 13:02:27.986269  8655 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:27.940724 11309 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:27.989580  9296 connection.cc:256] RPC call timeout handler was delayed by -2.2052973169999746s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18700000000000003s for a response
I20260709 13:02:27.995146  8656 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:27.998204  8729 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.976653 11364 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:27.971936 11377 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.999768  8548 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:28.006886  8727 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:27.980540 11189 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:27.965516 11262 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:02:28.011538  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:28.015125  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:28.018282  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2142) took 1529 ms. Trace:
W20260709 13:02:28.011888  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44083.9us
W20260709 13:02:28.016570  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69634.8us
I20260709 13:02:28.020028  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:27.980780 11358 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.002609 11283 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:28.022790  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 125825us
I20260709 13:02:28.018761  8728 rpcz_store.cc:276] 0709 13:02:26.488416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.326064 (+837648us) service_pool.cc:224] Handling call
0709 13:02:28.014923 (+688859us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:28.018199 (+  3276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3048320}
I20260709 13:02:27.995442 11276 raft_consensus.cc:2764] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:02:28.003927 11364 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.009749  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:28.072988  8656 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:28.039798  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:28.040131  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.086321  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:28.086660  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62678.3us
I20260709 13:02:28.088263  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:28.089756  8732 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.039991 11189 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.042529  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.043593  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.045902  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.011301 11389 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:28.121560  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:28.121517  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 168236us
I20260709 13:02:28.122799  8728 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.053709  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.123273  8657 raft_consensus.cc:2410] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:28.066537  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:28.070648  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.124219  9330 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:28.039166  8655 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:28.056773  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 69421.8us
I20260709 13:02:28.125483  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:28.130373  9329 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:28.131572  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:28.131872  8659 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:28.132442  8726 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:28.036105 11307 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:28.133417 11323 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:28.133773  8726 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:28.133857 11323 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:28.068166 11035 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:28.134533  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2141) took 1646 ms. Trace:
I20260709 13:02:28.135164 11035 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.135041 11323 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:28.134925  8726 rpcz_store.cc:276] 0709 13:02:26.488047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.133098 (+1645051us) service_pool.cc:224] Handling call
0709 13:02:28.133479 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:28.134301 (+   822us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.134483 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:28.136999 11323 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.011967 11377 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.025315 11279 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:28.137619  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 93977us
I20260709 13:02:28.137725 11236 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:28.138337 11260 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:28.139056  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 346001us
W20260709 13:02:28.137672  8697 leader_election.cc:336] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.544s (SENT)
W20260709 13:02:28.142109 11324 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.529s	user 0.004s	sys 0.005s
W20260709 13:02:28.186697  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.121517 11382 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:28.187469 11382 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:28.187709 11358 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:28.131904  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.188568  8654 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:28.142925  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:28.190364  8731 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:28.091810 11281 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 13:02:28.146098  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.146937  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:28.192417  8727 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.147723  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.133800 11307 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:28.150125 11345 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:28.068717 11364 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:28.206934  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 118815us
I20260709 13:02:28.155390  8732 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:28.156658  8729 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:28.087282 11300 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:28.166751  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:28.181985  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.183135  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.184269  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.185628  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:28.193506  9328 raft_consensus.cc:2410] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:28.150297 11276 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 13:02:28.137784  8725 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:28.186740  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 49120.8us
I20260709 13:02:28.124969 11384 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:28.185578 11260 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.200132 11189 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.200526  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.202427  8658 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:02:28.203101  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.249281  8650 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:28.203647  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.205291  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.205842  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.211004  8726 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:28.229954  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 546) took 1233 ms. Trace:
W20260709 13:02:28.231078  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.231001  8659 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:28.231485 11236 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:28.160696 11133 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.836s	user 0.004s	sys 0.010s
I20260709 13:02:28.255534  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.256290  9328 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:28.243512  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57953.2us
I20260709 13:02:28.198352 11220 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:28.246316  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.249506  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.261056  9325 raft_consensus.cc:2410] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:28.252498  8728 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:28.253726  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:28.234346  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.289157  8729 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:28.289428  9327 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:28.288580  8652 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:28.260388  8727 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:28.255286 11274 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 13:02:28.261168 11283 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:28.263077  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2146) took 1545 ms. Trace:
I20260709 13:02:28.298638  8726 rpcz_store.cc:276] 0709 13:02:26.717861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.137618 (+1419757us) service_pool.cc:224] Handling call
0709 13:02:28.210702 (+ 73084us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.250442 (+ 39740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.262812 (+ 12370us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b540032df50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:28.262997 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167205760}
I20260709 13:02:28.248471 11260 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:28.264777  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.265161 11393 tablet_replica.cc:333] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: stopping tablet replica
W20260709 13:02:28.265772  8697 connection.cc:256] RPC call timeout handler was delayed by -2.9840716500000326s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
W20260709 13:02:28.307149  8697 connection.cc:256] RPC call timeout handler was delayed by -2.9033158739999863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260709 13:02:28.307556  8697 connection.cc:256] RPC call timeout handler was delayed by -2.6612158739999927s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2041s for a response
I20260709 13:02:28.306066  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:28.307976  8697 connection.cc:256] RPC call timeout handler was delayed by -1.953509495999981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
W20260709 13:02:28.308369  8697 connection.cc:256] RPC call timeout handler was delayed by -1.8473094959999798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
W20260709 13:02:28.308673  8697 connection.cc:256] RPC call timeout handler was delayed by -1.7591094960000078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260709 13:02:28.308986  8697 connection.cc:256] RPC call timeout handler was delayed by -1.6187094960000081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
I20260709 13:02:28.265957 11054 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:28.266137  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.223304 11300 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:28.266680  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.268510  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1692) took 4121 ms. Trace:
I20260709 13:02:28.310061  8651 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.270290  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.270599 11377 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:28.271157 11309 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:28.270987 11236 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.271953 11279 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:28.235869 11345 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:28.332113  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 78469.2us
I20260709 13:02:28.310428  8577 rpcz_store.cc:276] 0709 13:02:24.147504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.895194 (+747690us) service_pool.cc:224] Handling call
0709 13:02:24.895499 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.246243 (+2350744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.268288 (+1022045us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b54000b6654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:28.268448 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5368564608}
I20260709 13:02:28.273723  9330 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:28.274942  8657 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:28.250850  8732 rpcz_store.cc:276] 0709 13:02:26.978333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.210584 (+1232251us) service_pool.cc:224] Handling call
0709 13:02:28.210956 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.211173 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:28.211209 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:28.211263 (+    54us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:28.211313 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.211397 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:28.211536 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:28.240639  9323 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:28.295310  8727 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:28.296635  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 104539us
W20260709 13:02:28.305784  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 139066us
I20260709 13:02:28.311719  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.339357  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:28.314576  9296 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.870s (SENT)
I20260709 13:02:28.325735  9326 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:28.328811  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.313519  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.339520  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:28.309662 11283 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:28.351649  8654 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:28.352348  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.343568  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:28.344383  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.344936  9323 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:28.346092 10816 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:28.345849 11345 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:28.347221  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 89426.6us
I20260709 13:02:28.343261  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.311233 11054 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:28.314142 11220 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:02:28.354512  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.355134  8727 raft_consensus.cc:2376] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 5. Re-sending same reply.
I20260709 13:02:28.354377  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:28.363641  8732 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:28.356523  9328 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:28.364709  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:28.357445  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.356294  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.359496  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:28.365590  8657 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:28.350288 11279 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:28.368515  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.371572 11054 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.372967 11054 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:28.373593 11397 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:28.374092 11397 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.374434 11397 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.375144 11300 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.376237 11300 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.378309 11279 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:28.369058  8658 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 4. Re-sending same reply.
W20260709 13:02:28.390080  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:28.398025  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44032us
I20260709 13:02:28.389678  8650 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:28.390432  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 81465.6us
W20260709 13:02:28.415280  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50644.6us
I20260709 13:02:28.416992  9329 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:28.417913  8652 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:28.418422  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:28.401230  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:28.419688  8727 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:28.419773  8732 raft_consensus.cc:2410] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 13:02:28.402941  8697 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.691s (SENT) [suppressed 52 similar messages]
I20260709 13:02:28.363936 11276 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:28.414481 11281 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:28.430891 11222 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:28.390298  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.414989 11035 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.434531  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2149) took 1674 ms. Trace:
I20260709 13:02:28.434916  8731 rpcz_store.cc:276] 0709 13:02:26.759667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.191078 (+1431411us) service_pool.cc:224] Handling call
0709 13:02:28.433748 (+242670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.434132 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:28.434189 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:28.434248 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:28.434283 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.434342 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:28.434449 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":906,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92}
W20260709 13:02:28.436148 11381 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.702s	user 0.002s	sys 0.000s
I20260709 13:02:28.440706  8655 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:28.443760  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.391013 11393 raft_consensus.cc:2243] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:02:28.446178  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:28.458196  9326 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:28.445282  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:28.460397  8727 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:28.465092  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:28.465984  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:28.469326  8582 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:28.444308 11276 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:28.470023  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:28.472872  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.473354  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.475800  8650 raft_consensus.cc:2368] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6.
I20260709 13:02:28.474606  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.475783  8655 raft_consensus.cc:2368] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5.
I20260709 13:02:28.477622  8727 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:02:28.481088  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 121656us
I20260709 13:02:28.481979  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:28.483246  9326 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:28.483726  8732 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:02:28.489112  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 86183.8us
I20260709 13:02:28.489874  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.491158  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:28.485733  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:28.492669  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:28.493305  8650 raft_consensus.cc:2368] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7.
I20260709 13:02:28.495524  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 548) took 1201 ms. Trace:
I20260709 13:02:28.495932  8729 rpcz_store.cc:276] 0709 13:02:27.293845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.212037 (+918192us) service_pool.cc:224] Handling call
0709 13:02:28.288983 (+ 76946us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.289594 (+   611us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.495313 (+205719us) spinlock_profiling.cc:243] Waited 76.5 ms on lock 0x7b5400229f50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:28.495434 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":175970176}
I20260709 13:02:28.496358  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:28.498351  8654 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 13:02:28.499596  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 529) took 1261 ms. Trace:
I20260709 13:02:28.499665  8651 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:28.501204  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.474658 11276 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.391259 11283 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:28.478760 11309 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:28.477166 11377 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:28.497280 11281 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:28.466565 11220 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 13:02:28.515232  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:28.509859  8727 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:28.510447  9326 raft_consensus.cc:2410] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:28.511623  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:28.517189  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.516012  8580 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:28.521648  8697 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.772s (SENT)
I20260709 13:02:28.520071  8652 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:28.524109  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:28.499996  8578 rpcz_store.cc:276] 0709 13:02:27.238296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.238528 (+   232us) service_pool.cc:224] Handling call
0709 13:02:27.266553 (+ 28025us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:27.305851 (+ 39298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:27.305885 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:27.319978 (+ 14093us) log.cc:844] Serialized 52 byte log entry
0709 13:02:27.485224 (+165246us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:27.485270 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.485360 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:28.435229 (+949869us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:28.486514 (+ 51285us) raft_consensus.cc:1685] finished
0709 13:02:28.486534 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:28.499548 (+ 13014us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1721,"raft.queue_time_us":7546,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":6086,"spinlock_wait_cycles":6802304,"thread_start_us":76738,"threads_started":1,"wal-append.queue_time_us":361296}
I20260709 13:02:28.507664 11406 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:28.509570 11160 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:28.510870 11377 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:28.469434 11238 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20260709 13:02:28.513638  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.533866  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:28.510741 11309 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.474491 11382 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.557595 11377 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:28.558257 11035 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:28.554328  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.558789 11035 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:28.546986 11406 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:28.559736 11035 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.542198 11389 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:28.559755  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.544924  8727 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:28.560936 11035 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.547735  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.567304  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:28.567503  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.567850 11382 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.567955  8582 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:28.568157  9328 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:28.566265 11281 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.569939  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.570693  9328 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:28.564846  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.569113 11281 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:28.576501  8727 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20260709 13:02:28.534715 11400 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 13:02:28.577646  8655 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:28.550580  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.541538 11401 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:28.550258  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:28.576954  8727 pending_rounds.cc:85] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 2
W20260709 13:02:28.545715 11402 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.064s	user 0.002s	sys 0.000s
I20260709 13:02:28.557667 11393 raft_consensus.cc:2272] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:02:28.595640  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:28.604671  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 93325.1us
I20260709 13:02:28.557720 11309 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:28.596675  8727 pending_rounds.cc:107] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:02:28.608861  8727 raft_consensus.cc:2889] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:28.571647  9322 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:28.560658 11406 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:28.615566  8581 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:28.596840 11401 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.619135  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:28.627645  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.628588  8655 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:28.756349  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.637372  8728 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:28.647881  8697 leader_election.cc:336] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.041s (SENT)
I20260709 13:02:28.636945  9328 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:28.705524  8548 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.662s (SENT)
I20260709 13:02:28.648545  8580 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:28.597487  9325 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:28.761680  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.762264  9328 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:28.632100  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 104518us
I20260709 13:02:28.764161  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.766029  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.766780  8655 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:28.658222  8581 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20260709 13:02:28.838410  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:28.839185  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:28.839708  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.878046  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:28.892500  8698 leader_election.cc:336] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.907s (SENT)
I20260709 13:02:28.892860  9328 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:28.757915 11375 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 13:02:28.893787  8658 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:28.894039  8583 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:02:28.897590  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2308) took 2701 ms (client timeout 2063 ms). Trace:
I20260709 13:02:28.897796  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:28.617391 11401 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:28.901701  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 196196us
I20260709 13:02:28.974870  8655 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:28.903836  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 276200us
I20260709 13:02:28.904716  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:28.910071  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 151845us
I20260709 13:02:28.976011  8733 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:28.631923 11353 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.274s
I20260709 13:02:28.895843  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:28.907128 11054 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:02:28.936388  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.661644 11260 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:28.952272 11408 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 13:02:28.632172  8577 raft_consensus.cc:1813] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Handling vote request from an unknown peer 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:28.912113  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:28.973543  8653 rpcz_store.cc:269] 0709 13:02:26.181337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.190545 (+  9208us) service_pool.cc:224] Handling call
0709 13:02:28.882427 (+2691882us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b54004704d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:28.882554 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2083844864}
W20260709 13:02:28.986764  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 222908us
I20260709 13:02:29.002111  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:29.008198  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:29.008865  8653 raft_consensus.cc:2376] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 6. Re-sending same reply.
I20260709 13:02:29.008262  8577 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:29.015478  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.023200 11208 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:29.024540  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.025121 11279 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:29.026386 11408 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:29.026841 11408 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.026914 11260 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:29.027709 11408 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.023051  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:29.029278  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.029067 11408 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:29.029927  8659 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:29.030192  8729 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 3: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:29.041690  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:29.045595  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:29.056802  8733 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:29.057365  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 554) took 1452 ms. Trace:
I20260709 13:02:29.061810  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:29.063454  8653 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:29.064249  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:29.066062  8655 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:29.057801  8733 rpcz_store.cc:276] 0709 13:02:27.605236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.883278 (+1278042us) service_pool.cc:224] Handling call
0709 13:02:29.057312 (+174034us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:29.070861  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.071499  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:29.030210 11281 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:29.078673  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.079025  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:29.080555  8653 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:29.077314  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.082430  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.076211 11222 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.084254  8659 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:29.093951  8729 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260709 13:02:29.094489  8729 pending_rounds.cc:85] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:02:29.094877  8729 pending_rounds.cc:107] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260709 13:02:29.095170  8729 raft_consensus.cc:2889] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:29.109968  8724 raft_consensus.cc:2410] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:29.110188 11353 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:29.110456 11354 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.110707 11353 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:29.111058 11354 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.111685 11353 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.113682 11393 ts_tablet_manager.cc:1916] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:02:29.114863 11353 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:29.122913  8653 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:29.081238 11279 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:29.093987 11222 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:29.129819 11300 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.753s	user 0.013s	sys 0.000s
I20260709 13:02:29.068796 11405 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 13:02:29.117702 11281 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:29.138924 11427 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 13:02:29.130487 11279 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.146100 11279 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:29.152124 11416 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 13:02:29.154477  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:29.145440 11397 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.771s	user 0.005s	sys 0.008s
I20260709 13:02:29.145077 11307 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.159176  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1828) took 1264 ms. Trace:
W20260709 13:02:29.156559 11276 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.638s	user 0.006s	sys 0.006s
W20260709 13:02:29.172243  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 89886.6us
I20260709 13:02:29.159783  8731 rpcz_store.cc:276] 0709 13:02:27.894453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.436082 (+541629us) service_pool.cc:224] Handling call
0709 13:02:28.436415 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:28.436816 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:28.436865 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:28.471033 (+ 34168us) log.cc:844] Serialized 26 byte log entry
0709 13:02:28.620371 (+149338us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:28.952797 (+332426us) log.cc:844] Serialized 14 byte log entry
0709 13:02:28.953279 (+   482us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.953370 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:29.158855 (+205485us) raft_consensus.cc:1685] finished
0709 13:02:29.158887 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:29.159077 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1840,"prepare.queue_time_us":290574,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":2864,"raft.queue_time_us":526,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":6790,"thread_start_us":395633,"threads_started":2,"wal-append.queue_time_us":494284}
I20260709 13:02:29.175140  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:29.172609 11344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.203s	user 0.007s	sys 0.017s
I20260709 13:02:29.175654 11222 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:29.180936 11323 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.014s	user 0.007s	sys 0.007s
I20260709 13:02:29.170497 11307 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.194451  9323 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:29.195873 11307 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:29.161590 11260 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:29.168677 11276 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.197271  8725 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:29.197808  8725 pending_rounds.cc:85] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:29.198086  8725 pending_rounds.cc:107] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:29.198390  8725 raft_consensus.cc:2889] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:29.199087 11276 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:29.183158 11377 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.625s	user 0.011s	sys 0.000s
I20260709 13:02:29.199975 11377 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.201597 11354 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:29.201494 11401 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:29.189934 11364 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.950s	user 0.012s	sys 0.000s
I20260709 13:02:29.202747  8655 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
W20260709 13:02:29.184580 11382 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.616s	user 0.011s	sys 0.000s
W20260709 13:02:29.191277 11309 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.563s	user 0.010s	sys 0.000s
I20260709 13:02:29.161856 11397 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.202240 11377 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:29.191076  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2155) took 2272 ms (client timeout 2621 ms). Trace:
W20260709 13:02:29.233702  8724 rpcz_store.cc:269] 0709 13:02:26.919114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.315223 (+396109us) service_pool.cc:224] Handling call
0709 13:02:29.190814 (+1875591us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b5400065254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:29.190979 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4096488064}
I20260709 13:02:29.202919 11364 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.206103 11401 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:29.208252  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:29.208348 11260 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:29.212393  9329 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:29.213965 11397 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:29.214625 11309 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.222700  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:29.258124  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 734) took 1230 ms. Trace:
I20260709 13:02:29.226781 11262 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:29.230336  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:29.231204  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:29.204388 11344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.228s	user 0.007s	sys 0.017s
W20260709 13:02:29.250452  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 153750us
I20260709 13:02:29.258481  9331 rpcz_store.cc:276] 0709 13:02:28.027332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.222279 (+1194947us) service_pool.cc:224] Handling call
0709 13:02:29.258054 (+ 35775us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:29.259904  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.260098  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.261360  8726 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:02:29.262611  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.257428 11260 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.264204 11281 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:29.264321 11260 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.264479 11377 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:29.264904 11281 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.265220 11377 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:29.259382 11262 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.266331 11377 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.280648 11406 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:29.281114 11406 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:29.264321  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.282084 11406 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.283552 11406 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:29.265345  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.266788  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.267107  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:29.285450  8731 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:29.267802  8729 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:29.290189  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:29.291860  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.292695  8729 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:29.269268 11262 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.294771 11262 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:29.294855 11236 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:29.297499  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 726) took 3136 ms. Trace:
I20260709 13:02:29.267138 11434 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:29.270243  8724 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:29.269675  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.328797  8733 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
W20260709 13:02:29.277145  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 68954.9us
W20260709 13:02:29.268265  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.320607  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:29.278252  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.328408  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 551) took 1759 ms. Trace:
I20260709 13:02:29.333097  8576 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.333333  8728 rpcz_store.cc:276] 0709 13:02:27.574297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.288712 (+714415us) service_pool.cc:224] Handling call
0709 13:02:28.289142 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.759915 (+470773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.328157 (+568242us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b5400416254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:29.328346 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":800151808}
I20260709 13:02:29.298908  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.338133  8729 raft_consensus.cc:2410] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:29.338202  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:29.341295  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:29.327273 11220 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.304173 11307 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.298436 11434 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.276592 11309 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:29.348323  8727 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:29.348034  8731 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:29.320387 11398 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 13:02:29.285672 11353 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:29.352049  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:29.358385  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2157) took 2438 ms (client timeout 2946 ms). Trace:
I20260709 13:02:29.258701 11279 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.337006 11364 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:29.267622 11276 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
W20260709 13:02:29.358918  8730 rpcz_store.cc:269] 0709 13:02:26.919916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.321559 (+401643us) service_pool.cc:224] Handling call
0709 13:02:29.351934 (+2030375us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:29.358321 (+  6387us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3829120}
I20260709 13:02:29.360297 11222 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.337244 11430 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 13:02:29.298056  9324 rpcz_store.cc:276] 0709 13:02:26.161354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.213304 (+1051950us) service_pool.cc:224] Handling call
0709 13:02:27.230649 (+ 17345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.363716 (+1133067us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:28.366234 (+  2518us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:28.367739 (+  1505us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:28.402796 (+ 35057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.297194 (+894398us) spinlock_profiling.cc:243] Waited 961 ms on lock 0x7b5400693054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:29.297248 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:29.297430 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7686,"raft.queue_time_us":98002,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":2210110336,"thread_start_us":94327,"threads_started":1}
I20260709 13:02:29.352932 11344 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:29.366372 11133 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:29.369467 11220 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:29.266122 11281 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.371315  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:29.275913 11260 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.366417  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:29.377125  8576 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:29.380112  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 47858.2us
I20260709 13:02:29.381994  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:29.350351 11434 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.366400 11398 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:29.400278  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:29.411075  9330 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:29.398491  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:29.401975  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:29.412246  9326 raft_consensus.cc:2376] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3. Re-sending same reply.
I20260709 13:02:29.383711 11133 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.403975  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:29.413975  9296 leader_election.cc:400] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6.
I20260709 13:02:29.417840  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:29.418792  9323 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:29.419917 11440 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:29.420352 11440 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.420707 11440 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.419445 11133 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.421427 11133 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:29.423147  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.424396 11247 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:29.424834 11247 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.425966  9322 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:02:29.425686 11247 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.426988 11247 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:29.385316 11222 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.430154 11222 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.431828  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 550) took 1961 ms. Trace:
I20260709 13:02:29.432356  8725 rpcz_store.cc:276] 0709 13:02:27.470244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.271747 (+801503us) service_pool.cc:224] Handling call
0709 13:02:29.196827 (+925080us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.198914 (+  2087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.431590 (+232676us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b5400065250. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:29.431761 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2117220736}
I20260709 13:02:29.354477 11435 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20260709 13:02:29.435393  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 537) took 1009 ms. Trace:
I20260709 13:02:29.436633 11260 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:29.435827  8575 rpcz_store.cc:276] 0709 13:02:28.442566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.450827 (+  8261us) service_pool.cc:224] Handling call
0709 13:02:28.467199 (+ 16372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:28.467500 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:28.467540 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:29.220299 (+752759us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b2c000014c8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:02:29.221011 (+   712us) log.cc:844] Serialized 26 byte log entry
0709 13:02:29.349343 (+128332us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:29.349382 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.349479 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:29.435042 (+ 85563us) raft_consensus.cc:1685] finished
0709 13:02:29.435068 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:29.435325 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6810368,"thread_start_us":127884,"threads_started":1,"wal-append.queue_time_us":175866}
I20260709 13:02:29.436894 11408 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:29.437368 11260 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:29.437501 11408 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.438496 11408 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:29.454315  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 60674.2us
I20260709 13:02:29.459504  9296 leader_election.cc:400] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5.
W20260709 13:02:29.462081  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 95733.4us
I20260709 13:02:29.463320  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:29.384635 11401 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260709 13:02:29.409420 11398 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:29.464665  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 66276.6us
I20260709 13:02:29.468418 11249 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:29.456245 11377 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:29.512596 11422 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.575s	user 0.000s	sys 0.004s
I20260709 13:02:29.512228 11377 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.514515 11377 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:29.506237  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:29.534147  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66566.4us
W20260709 13:02:29.542207  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 84169.2us
W20260709 13:02:29.558933  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 95729.6us
I20260709 13:02:29.559901  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.466297 11353 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:29.635139  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 292208us
W20260709 13:02:29.637543  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 77995.6us
W20260709 13:02:29.642354  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.458782 11397 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.175s
W20260709 13:02:29.645929  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 86088.3us
I20260709 13:02:29.647528  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.655992  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:29.641995 11208 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.658866 11208 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.660087  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.633154 11451 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.563445 11344 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.094s
W20260709 13:02:29.663565  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 208044us
I20260709 13:02:29.664718  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.665647  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.667457  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:29.667801  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:29.668244  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:29.668653  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.670333  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.671904  9296 leader_election.cc:400] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7.
I20260709 13:02:29.672487  9296 leader_election.cc:403] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:02:29.677620 11260 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:29.678465 11260 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.679015 11260 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:29.680711 11208 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:29.683943 11397 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7.
I20260709 13:02:29.661895 11437 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:29.663405 11406 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:29.665982 11281 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:29.665958 11279 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:29.670477 11323 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:29.703248 11208 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:29.703812  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:29.670236 11357 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:02:29.704855 11357 consensus_queue.cc:1243] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:29.698956 11281 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:29.703840  9322 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:29.707684  9327 raft_consensus.cc:2410] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:02:29.708257 11393 ts_tablet_manager.cc:1929] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:02:29.708824 11393 log.cc:1199] T 2947646ebfc34bd7a5a55312f5812ee2 P a0d703ee18464475a290425976246555: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/wals/2947646ebfc34bd7a5a55312f5812ee2
I20260709 13:02:29.673427 11451 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.709980 11208 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:29.710435 11208 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:29.710189 11451 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.711324 11208 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.712540 11208 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:29.705772 11281 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:29.716583  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.716753  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 743) took 1566 ms. Trace:
W20260709 13:02:29.717489  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.717309  9327 rpcz_store.cc:276] 0709 13:02:28.149967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.337053 (+187086us) service_pool.cc:224] Handling call
0709 13:02:29.714997 (+1377944us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54007ab554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:29.716690 (+  1693us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3082854656}
I20260709 13:02:29.721812  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 556) took 2115 ms. Trace:
I20260709 13:02:29.722250  8726 rpcz_store.cc:276] 0709 13:02:27.605974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.300045 (+694071us) service_pool.cc:224] Handling call
0709 13:02:28.311280 (+ 11235us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:29.263897 (+952617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.716093 (+452196us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b5400472cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:29.721749 (+  5656us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2183751040}
I20260709 13:02:29.692652 11307 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:29.725556  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:29.726052  9297 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.899s (SENT)
I20260709 13:02:29.726203  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:29.724988  8623 connection.cc:256] RPC call timeout handler was delayed by -1.2777884189999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
I20260709 13:02:29.726596  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:29.727332  8623 connection.cc:256] RPC call timeout handler was delayed by -1.0392884190000018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
W20260709 13:02:29.727902  8623 connection.cc:256] RPC call timeout handler was delayed by -1.035688418999996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
W20260709 13:02:29.728279  8623 connection.cc:256] RPC call timeout handler was delayed by -1.0212884189999727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20260709 13:02:29.728785  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.726413  9327 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:29.730962  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.732168  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.732999  8733 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:29.733111  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:29.734627  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.725651  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:29.663566 11220 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.737028  9325 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:29.737179 11440 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.727335 11399 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.738206  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.727269 11445 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 13:02:29.727190 11409 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.718772 11449 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:29.739593 11440 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:29.693930 11398 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.749090  8729 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:29.739816  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.716358 11160 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:29.740512  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.740725  9322 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:29.741183  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.695303 11353 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.742878 11323 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:29.743294 11399 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:29.744259  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:29.744719 11434 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.745343  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.810942  8726 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 2.
I20260709 13:02:29.745843 11377 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:29.727136 11443 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:29.748400  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.750901  8724 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:29.754032  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.753528  8727 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:29.755895  8730 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:29.822419  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 393) took 1342 ms. Trace:
W20260709 13:02:29.822695  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 68720.7us
I20260709 13:02:29.757145  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:29.754812  9322 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:29.758373  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 539) took 1218 ms. Trace:
I20260709 13:02:29.759335  8733 raft_consensus.cc:2376] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 4. Re-sending same reply.
I20260709 13:02:29.832836  8580 rpcz_store.cc:276] 0709 13:02:28.539624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.545537 (+  5913us) service_pool.cc:224] Handling call
0709 13:02:28.547341 (+  1804us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.781643 (+234302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.758113 (+976470us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54002d87d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:29.758310 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232115072}
I20260709 13:02:29.833777  8733 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:29.835301  8733 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:29.772104 11160 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.750501 11449 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:29.780514  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.780457  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:29.772468 11399 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.782137 11430 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:29.783564  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:29.756651 11446 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:29.784857 11307 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:29.792021 11417 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.652s	user 0.004s	sys 0.008s
I20260709 13:02:29.792825 11393 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41978 (request call id 84) took 2227 ms. Trace:
I20260709 13:02:29.793655 11398 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:29.754892 11409 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.776016  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 742) took 1286 ms. Trace:
I20260709 13:02:29.898257  9329 rpcz_store.cc:276] 0709 13:02:28.489327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.489588 (+   261us) service_pool.cc:224] Handling call
0709 13:02:28.489884 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.251820 (+761936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.775688 (+523868us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b54002148d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:29.775940 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1660879232}
I20260709 13:02:29.822008  9327 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:29.824712  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:29.824923  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.826813  8730 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:29.822888  8658 rpcz_store.cc:276] 0709 13:02:28.479846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.479972 (+   126us) service_pool.cc:224] Handling call
0709 13:02:28.882753 (+402781us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.972315 (+ 89562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.822051 (+849736us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54004704d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:29.822351 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":900178304}
W20260709 13:02:29.831765  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.837389  8733 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:29.871281 11212 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.841622  8731 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:29.871374 11449 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:29.911839 11435 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:29.816668 11377 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.913163 11377 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.878759  8727 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:29.880307  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:29.918802  8729 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:29.882007  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 206104us
I20260709 13:02:29.836272 11208 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.922021 11208 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.923593 11249 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:29.924219 11249 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:29.925714  8580 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 13:02:29.903857  8733 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:29.926672 11357 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:02:29.837740  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 167465us
I20260709 13:02:29.872085 11364 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.840251 11443 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:29.928762  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.876228 11430 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:29.938657  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.937971  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:29.940132  8733 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:29.945171  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 202652us
W20260709 13:02:29.945968  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 278809us
W20260709 13:02:29.945895  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 280149us
I20260709 13:02:29.946947  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:29.905759 11446 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:29.948146  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:29.949968  8548 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.124s (SENT)
I20260709 13:02:29.948041  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.938273 11357 consensus_queue.cc:1243] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:29.966563  8731 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:29.969561  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.970057  8583 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:29.973915  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.952433  8625 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.984213  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:29.984362  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:29.985435  8576 raft_consensus.cc:2410] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:29.987880  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:29.988073  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:29.988605  8579 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:30.001531  8581 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:29.985252  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:29.980789  8729 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:30.001216  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.934528 11212 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.006484  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:30.006691  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.020140  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.007941  8580 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:30.022460 11455 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.023599 11455 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:30.010205  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 183428us
I20260709 13:02:30.025542  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:30.027200  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.027572  8733 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:30.028401  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.938098 11364 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.032853  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:29.983899 11208 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.040278  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:30.038808  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 84010.4us
I20260709 13:02:30.041610  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1832) took 2145 ms. Trace:
I20260709 13:02:30.051165  8575 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.051301  8732 rpcz_store.cc:276] 0709 13:02:27.895848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.484497 (+588649us) service_pool.cc:224] Handling call
0709 13:02:28.505041 (+ 20544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:28.522068 (+ 17027us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:28.522118 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:28.522778 (+   660us) log.cc:844] Serialized 26 byte log entry
0709 13:02:28.615699 (+ 92921us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:28.615736 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:28.615808 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:29.791282 (+1175474us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:30.033532 (+242250us) raft_consensus.cc:1685] finished
0709 13:02:30.033551 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:30.041513 (+  7962us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9899,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":23101,"spinlock_wait_cycles":9925888,"thread_start_us":73661,"threads_started":1,"wal-append.queue_time_us":593937}
I20260709 13:02:29.987478  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:30.055502  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 108600us
I20260709 13:02:30.042114  8727 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:30.057757  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:30.047751  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:30.004597 11249 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.061054  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:29.972021 11281 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:29.937731  9322 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.032686 11344 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:30.069821  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 81804.6us
I20260709 13:02:30.029963  8729 raft_consensus.cc:2376] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6. Re-sending same reply.
I20260709 13:02:30.071893  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.067507  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:30.024973 11357 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 13:02:30.037819 11276 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:30.078123 11353 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 13:02:30.039341 11405 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:30.133186  8575 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
W20260709 13:02:30.074247  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 117500us
I20260709 13:02:30.096580  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:30.102268  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:30.105222  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.034624 10816 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:30.134574  8733 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:30.072911 11435 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:30.120750 11276 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:30.124759  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:30.121580  8724 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 13:02:30.152153  8727 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:30.077554 11332 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:30.155555  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 97883.2us
I20260709 13:02:30.124012 11439 raft_consensus.cc:2764] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:02:30.130767  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.131104  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 540) took 1560 ms. Trace:
I20260709 13:02:30.125627 11208 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:30.131496  8732 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:30.131532  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:30.131992  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.134574  8579 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:30.175429  8580 raft_consensus.cc:2410] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:02:30.115221  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.120090 11446 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.141966  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.140118 11434 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.142891  8578 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:30.137900 10816 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:30.148880  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.149175  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.084750 11447 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:30.038578 11430 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.083096 11220 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:30.103021 11344 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.159459  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:30.159766  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92325.2us
I20260709 13:02:30.055035 11260 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.162075  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.068382 11323 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260709 13:02:30.130702  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.202258  8651 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.164883  8658 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:30.167451  8653 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:30.168751  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.164965  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:30.204826  8728 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.143119 11274 raft_consensus.cc:2764] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:02:30.138602 11405 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.165438  8582 rpcz_store.cc:276] 0709 13:02:28.570273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.570590 (+   317us) service_pool.cc:224] Handling call
0709 13:02:28.593105 (+ 22515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.593325 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:29.742744 (+1149419us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b1800359aa0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf
0709 13:02:29.742906 (+   162us) log.cc:844] Serialized 14 byte log entry
0709 13:02:30.101746 (+358840us) log.cc:844] Serialized 14 byte log entry
0709 13:02:30.122126 (+ 20380us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:30.122202 (+    76us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:30.130860 (+  8658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.130940 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:30.131049 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12348,"prepare.queue_time_us":327012,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":37837,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":23863,"spinlock_wait_cycles":7570688,"thread_start_us":737112,"threads_started":3,"wal-append.queue_time_us":330483}
I20260709 13:02:30.137055 11222 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:30.072841 11454 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:30.243813  8651 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:30.154577 11276 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:30.177234  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.184621 11447 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.187263  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:29.897994 11393 rpcz_store.cc:276] 0709 13:02:27.640119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.689480 (+ 49361us) service_pool.cc:224] Handling call
0709 13:02:28.061806 (+372326us) ts_tablet_manager.cc:1151] Deleting tablet 2947646ebfc34bd7a5a55312f5812ee2
0709 13:02:28.061949 (+   143us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 13:02:28.265008 (+203059us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54003214d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1eff9 00007f140ec2c482 00007f140ec5d2d0 00007f140ec4383d 00007f140ec437e9 00007f140ec43779 00007f140ec43741 00007f140ec4296d 000000000038f354 000000000038f199
0709 13:02:29.276281 (+1011273us) tablet_metadata.cc:665] Metadata flushed
0709 13:02:29.708113 (+431832us) tablet_metadata.cc:665] Metadata flushed
0709 13:02:29.792693 (+ 84580us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4763,"spinlock_wait_cycles":265289472,"tablet-delete.queue_time_us":289915,"thread_start_us":200401,"threads_started":1}
I20260709 13:02:30.194023  8575 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:30.197759  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.198050  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.199550  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.224354 11409 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.185918 11469 raft_consensus.cc:2764] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:02:30.218762 10816 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.217758 11405 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:30.217123 11451 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:30.190821 11160 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.149962 11456 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20260709 13:02:30.238539  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 123411us
I20260709 13:02:30.225268  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.241331 11212 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.242024  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.242619  8725 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:30.197678 11344 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.247676  8727 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:30.250969  8658 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260709 13:02:30.262109 11364 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.263444 11377 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.283936  8578 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:30.264263 11323 consensus_queue.cc:1243] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:30.239051 11439 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 13:02:30.101686 11443 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.209306 11430 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:30.265496 11407 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 13:02:30.266325 11447 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:30.269552  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 106221us
W20260709 13:02:30.247823 11462 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.097s	user 0.002s	sys 0.000s
I20260709 13:02:30.244277 11454 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.288533  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:30.273031  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 247572us
I20260709 13:02:30.283464  8580 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:30.290916  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.307862  8659 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 2.
I20260709 13:02:30.394371  8656 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:30.321219  8728 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:30.413383  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:30.323882  8579 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:30.393880  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 237236us
W20260709 13:02:30.322235  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 167825us
I20260709 13:02:30.420099  8727 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 13:02:30.420419  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.422612  8727 pending_rounds.cc:85] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 3
I20260709 13:02:30.422909  8727 pending_rounds.cc:107] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 13:02:30.423039  8578 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:30.423241  8727 raft_consensus.cc:2889] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:30.306020 11463 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.096s	user 0.001s	sys 0.002s
I20260709 13:02:30.289059 11160 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.417996 11220 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:30.471865 11260 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:30.359678 11470 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 13:02:30.359712 11479 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.303089 11212 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:30.284967 11409 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.187s
W20260709 13:02:30.478168  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 61305.8us
W20260709 13:02:30.478953  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56110.3us
W20260709 13:02:30.480618 11457 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 13:02:30.377967 11466 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.431125 11274 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.360205 11323 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:30.284053 11435 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.473136 11364 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
W20260709 13:02:30.490553  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 77108.9us
I20260709 13:02:30.347272 10816 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.314016 11451 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:30.289539 11353 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.494244 11473 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:30.494854 11377 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:30.501629 11418 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.481s	user 0.017s	sys 0.006s
I20260709 13:02:30.509718  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:30.513530  8698 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.049s (SENT)
I20260709 13:02:30.416706 11454 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:30.554934 11323 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:30.531059  8578 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:02:30.529808 11467 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 13:02:30.533834 11472 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.534029  9297 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:30.534354 11484 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.590471 11484 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.534893 11481 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 13:02:30.536615 11332 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:30.495045 11476 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.594098 11476 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:30.595038 11449 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.654s	user 0.006s	sys 0.005s
I20260709 13:02:30.595564 11449 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.596935  8657 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:30.597100 11476 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:30.544214 11474 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260709 13:02:30.545121 11477 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:30.602713 11477 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:30.603641 11477 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.604908 11477 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:30.545320 11220 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:30.549969  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 354080us
I20260709 13:02:30.549067  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.572556  8581 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:30.504479 11399 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
W20260709 13:02:30.575709  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62213.9us
W20260709 13:02:30.575748  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66407.3us
I20260709 13:02:30.571832 11473 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.554562 11383 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 13:02:30.571671 11469 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
W20260709 13:02:30.589775  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 55808.7us
I20260709 13:02:30.513559 11407 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 13:02:30.627890  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.594092 11481 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:30.598454 11447 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.525332 11160 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.600617 11405 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.593309 11484 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.593822 11439 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:30.617799  8698 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.135s (SENT)
I20260709 13:02:30.640326  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.623683 11473 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:30.619992  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:30.625389  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 79773.7us
I20260709 13:02:30.645059  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.585034 11446 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:30.589062 11276 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:30.635623 11460 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 13:02:30.632087 11222 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:30.526149  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 235265us
I20260709 13:02:30.531984 11479 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.572065 11353 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.622359 11332 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.657176 11332 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.645715 11451 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.659092 11483 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:30.659731 11332 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.660322 11260 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:30.661695 11383 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.662851 11479 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.629541 11472 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.664697 11353 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.664809 11472 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.666610 11160 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.666839  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:30.649290 11344 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.667953  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:30.668084 11447 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.668824  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:30.623250 11467 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 13:02:30.556236 11466 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.647651  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.619189  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:30.670779  8575 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:30.642304  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.671416  8581 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:30.649452  8576 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:30.633044 11481 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:30.652652  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:30.630630 11476 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 13:02:30.673280 11437 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:30.674278  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.675529  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:30.642755 11469 consensus_queue.cc:1243] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:30.679415  8577 raft_consensus.cc:2410] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:02:30.682816  8579 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:02:30.684178 11440 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:30.684556 11396 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:30.681254 11279 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:30.660784 11480 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 13:02:30.693697 11449 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.696985 11405 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.698792 11276 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:30.698984  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.647904 11439 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:30.699186 11466 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.669184 11344 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.676738 11446 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:30.706043  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:30.706823 11332 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:30.708271  8575 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:30.708091  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 88993us
W20260709 13:02:30.708173  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55597.8us
I20260709 13:02:30.709159  8580 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:02:30.730016  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:30.730398  9327 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:30.714841 11396 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:30.731935 11407 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:30.710922 11398 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.800s	user 0.012s	sys 0.000s
I20260709 13:02:30.717932 11279 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:30.724601  8581 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:02:30.725806  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.715098 11447 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:30.719007 10816 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 13:02:30.714689 11383 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:30.653133 11484 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:30.731478 11396 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.726994 11486 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:30.725240 11494 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.738749  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:30.738830 11494 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.739449  9328 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 3.
I20260709 13:02:30.734114 11279 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.711890 11430 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 13:02:30.729022 11399 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.727535 11473 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.745363  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:30.752071  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:30.738265 11396 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:30.743214 11383 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.758462 11478 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.758805 11439 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.759341  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:30.759111 11478 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.760227 11383 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.735903 11466 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 13:02:30.764029 11481 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.765151 11454 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.766871 11485 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:30.767323 11485 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:30.767653 11454 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.768314  9325 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:30.768715 11446 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.769580 11364 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.769536  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:30.771373 11364 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.771965  9325 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:30.772905 11494 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.737032 11407 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.773710  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:30.774605  9323 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:30.758473  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:30.769694 11160 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.778251 11160 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.757409 11399 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.784003 11399 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.743244 10816 consensus_queue.cc:1243] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:30.754113  9326 raft_consensus.cc:2376] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 4. Re-sending same reply.
I20260709 13:02:30.788465 11383 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.754884  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:30.790169  9330 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.790959 11473 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.756220  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:30.794425  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49109.4us
W20260709 13:02:30.795195  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:30.796095  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.796907  8697 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:30.799178  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:30.741950 11486 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:30.743712 11471 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:30.806061 11472 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:30.816246  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:30.816854  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.821249  8697 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.804741 11447 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.805330 11160 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.778255 11407 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:30.789156 11383 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.825043  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:30.822957  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:30.832916  9330 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:30.835920  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:30.822676 11481 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:30.823122 11479 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:30.825745 11469 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.825671 11364 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:30.844859  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:30.822885 11488 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.768316 11485 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.848227 11488 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:30.848593 11486 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.849747 11488 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:02:30.850605 11353 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.845779  9328 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:30.837285 11160 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.823028 11449 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 13:02:30.839641 11471 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:30.865836  9329 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:30.839325  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:30.841593 11473 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:30.870000  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.853978 11446 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260709 13:02:30.873221 11478 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:30.872330 11473 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.875391 11473 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.876652 11279 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.878360 11471 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:30.881232  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:30.882944  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:30.886941 11478 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:30.879807 11471 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:30.887634 11364 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:30.888074 11364 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.888437 11364 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:30.891088  9296 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.767s (SENT)
I20260709 13:02:30.893934  9296 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:30.891584  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:30.892154  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:30.867051 11479 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:30.895349  8729 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:30.902474  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:30.903375  8726 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:30.904692  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:30.907931  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:30.909190  8732 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:30.888774 11160 raft_consensus.cc:2764] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:02:30.934863 11367 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:30.935606 11367 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:30.937745 11279 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:30.942493  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 365153us
I20260709 13:02:30.947206  8651 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:30.898684 11478 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:30.951093  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.951406  8651 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:30.951719  8657 raft_consensus.cc:2376] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6. Re-sending same reply.
I20260709 13:02:30.944993  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.952422  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.915663 11399 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:02:30.953490  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.953675  8651 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:30.954355  8655 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:30.954854  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.961540  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.962726  8653 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:30.961652  8650 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:30.965355  8659 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:30.965683  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:30.967252  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.969247  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.967340 11478 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:30.970273  8657 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:30.970193 11472 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:30.971377  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:30.971309  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.972900  8655 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:30.976075  8658 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:30.977427  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.972855  8726 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:30.978447  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.982895  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.983589  8651 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:30.978961  8658 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:30.987743  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.988305  8658 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:30.989362  8726 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:30.989292  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.990643  8650 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:30.961071 11208 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.026s	sys 0.033s
I20260709 13:02:30.996786  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:30.997478  8655 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:30.999166  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:30.999621  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.007490  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:31.000085 11208 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.026s	sys 0.033s
I20260709 13:02:30.941898 11367 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.015936  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2192) took 1028 ms. Trace:
I20260709 13:02:31.021332 11507 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.001799 11455 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.029429 11455 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.030882  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:31.014669 11440 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:31.023146  8731 rpcz_store.cc:276] 0709 13:02:29.988002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.988868 (+   866us) service_pool.cc:224] Handling call
0709 13:02:29.989178 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:30.879493 (+890315us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b18003b1400. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f1409c0e060 00007f1409c0ebfb 00007f1409c1964b
0709 13:02:30.879614 (+   121us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:30.879645 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:30.880108 (+   463us) log.cc:844] Serialized 26 byte log entry
0709 13:02:30.912237 (+ 32129us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:30.912285 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.912473 (+   188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:31.010788 (+ 98315us) raft_consensus.cc:1685] finished
0709 13:02:31.010809 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:31.015849 (+  5040us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":69937,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":14177,"spinlock_wait_cycles":12432128,"thread_start_us":123704,"threads_started":2,"wal-append.queue_time_us":81024}
I20260709 13:02:31.038537  8733 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:31.013316 11208 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:31.026142 11507 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.042541  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:31.043358  9326 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:31.044291 11507 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:31.049437 11377 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:31.049974 11377 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:31.050256 11473 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.042596  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:31.051445  9327 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.050891 11377 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.029232 11332 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:31.053015  8653 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:31.052860 11377 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.087718 11332 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.091358 11281 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:31.091797 11364 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.031739 11455 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:31.094293  8730 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:31.096830  9327 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:31.097486  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:31.098317  8655 raft_consensus.cc:2376] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5. Re-sending same reply.
I20260709 13:02:31.098564  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.098277 11383 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:31.101694  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:31.103602  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:31.104457  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:31.105459  9326 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:31.105264  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:31.107636 11383 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.107539  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:31.109380  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:31.098121 11367 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:31.105491 11455 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:31.113862 11281 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:31.117686 11514 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:31.118238 11514 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.118608 11514 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:31.120582  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.121021  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.126713  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.131330  9326 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:31.133576 11262 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.133776 11450 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.134002 11208 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.134642 11450 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.136544  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:31.011442 11479 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.134253 11262 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.153963  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:31.142506  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.149658  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:31.150882  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.145576  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:31.155697  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46508.5us
I20260709 13:02:31.152154  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:31.154784  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:31.156781  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:31.159361  8733 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:31.108307 11383 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.162598  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:31.119707 11473 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:31.139134 11515 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.062s	user 0.002s	sys 0.000s
I20260709 13:02:31.163291  8724 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 13:02:31.171056  8724 pending_rounds.cc:85] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 3
I20260709 13:02:31.171324  8724 pending_rounds.cc:107] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 13:02:31.171766  8724 raft_consensus.cc:2889] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:31.165814  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.138899 11450 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:31.173910  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 119488us
I20260709 13:02:31.177057  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.212687  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:31.219141  8731 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:31.213563  8577 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:31.171211  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.153801  9322 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.217192  8729 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:31.220676  8575 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:31.221935  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.223601  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.221269  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.225314  9326 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:31.173683 11472 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:31.225728  8584 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:31.163075 11516 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:31.233649  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:31.239379  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.166807 11276 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:31.164357 11397 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:31.217698 11494 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.235690  8575 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:31.234824  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.172821 11478 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 13:02:31.217175 11260 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:31.243647 11510 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260709 13:02:31.243963  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:31.261148  8725 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:31.249817 11504 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.251178 11469 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:31.236002 11345 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.271572 11507 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:31.255295 11473 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:31.224452 11160 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:31.227172 11440 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.265844  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:31.268348  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:31.269227  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.213992 11407 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20260709 13:02:31.289108  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.244483 11500 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:31.294418  8731 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:31.235405 11513 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:31.228617 11279 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260709 13:02:31.306052  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.275578 11516 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:31.347193  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.281826 11276 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:31.290606 11481 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:31.352882  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.289796 11260 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.317637 11507 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.306002 11469 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:31.306174 11472 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.374986  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.284653 11494 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.375691  8580 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:31.306528 11449 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260709 13:02:31.373512  8584 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:31.379451  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.347589 11345 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.361847 11407 consensus_queue.cc:1243] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:31.357062 11513 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:31.396135  8581 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:31.399096  8654 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:31.400807 11276 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.401831 11345 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:31.378083 11507 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.352759 11160 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.407483  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.316409 11504 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.417100  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:31.407387  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.425989  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:31.427793  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.435954  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.437053  8584 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.445277  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.446244  8582 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:31.447125  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.395251 11472 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.433833 10816 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.461555  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.462451  8582 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.449136  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.438998 11471 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:31.465292  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.471385  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.460213 11377 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:31.474280 11513 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:31.475304 11518 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:02:31.450116 11383 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:31.444341 11516 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.479864 11260 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:31.459844 11279 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.480229  8584 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:31.481518  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.475327  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.482657  8584 raft_consensus.cc:2410] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:02:31.413977 11469 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.497193  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.484705 11377 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:31.502787 11472 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:31.502403 11377 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.480700 11279 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.465428 11471 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.513798  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.510509  8577 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:31.506300 11504 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:31.494673 11383 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.512213 11160 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:31.470482 10816 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.509837  8581 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:31.515895  8577 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:31.516486  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.521600 11449 consensus_queue.cc:1243] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:31.512243 11279 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:31.525941  8579 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 6 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:31.522918  9322 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260709 13:02:31.526881  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.527876  8579 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:31.533015  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.517071 11471 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:31.535365  8578 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:31.531354  9322 pending_rounds.cc:85] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 6
I20260709 13:02:31.526028 11383 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.538550  8577 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:31.525053 11469 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.540652 11494 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:31.540920 11523 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 13:02:31.539366  9322 pending_rounds.cc:107] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 4.7
W20260709 13:02:31.541617 11520 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:02:31.543946  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:31.551703  9330 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:31.553117 11484 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:31.553663  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:31.550596  9322 raft_consensus.cc:2889] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:31.553892  8584 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:31.559199 11323 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:31.574802 11324 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 13:02:31.570190 11484 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:31.575209 11323 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:31.562760 10816 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:31.575989 11484 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.562935  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:31.577863  8583 raft_consensus.cc:2376] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8. Re-sending same reply.
W20260709 13:02:31.578364  8583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1717) took 1595 ms (client timeout 1975 ms). Trace:
W20260709 13:02:31.578691  8583 rpcz_store.cc:269] 0709 13:02:29.982809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.562675 (+1579866us) service_pool.cc:224] Handling call
0709 13:02:31.578309 (+ 15634us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:31.564958  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:31.580277 11409 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:31.580895 11409 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:31.581589 11279 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:31.582006 11279 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.582386 11279 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.584734  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2218) took 1446 ms. Trace:
I20260709 13:02:31.585157  8656 rpcz_store.cc:276] 0709 13:02:30.144171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.153998 (+  9827us) service_pool.cc:224] Handling call
0709 13:02:31.584534 (+1430536us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400420c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:31.584676 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":255824256}
I20260709 13:02:31.568997  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:31.587891  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:31.570550  9330 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:31.571193  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.571607 11516 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.574327  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 755) took 1859 ms. Trace:
I20260709 13:02:31.580341  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:31.591471  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:31.595144  8582 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:31.587217 10816 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:31.574865 11383 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:31.596525 11494 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:31.596957 11494 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.597721 11281 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:31.598246 11281 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:31.596400  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:31.599177 11281 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.599895  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:31.577246 11484 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:31.601099  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:31.604363  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:31.610376  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.611068  8728 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:31.597301 11494 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:31.603355 11398 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:31.618016  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:31.601709 11450 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260709 13:02:31.601576 11260 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 13:02:31.615222 11383 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:31.628793 11450 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:31.245337  8466 catalog_manager.cc:5028] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): tablet 2947646ebfc34bd7a5a55312f5812ee2 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
I20260709 13:02:31.641167 11212 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:31.644457 11262 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:31.596392  9322 rpcz_store.cc:276] 0709 13:02:29.715296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.721238 (+  5942us) service_pool.cc:224] Handling call
0709 13:02:29.740539 (+ 19301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.037571 (+1297032us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b58005702d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b720aa 00007f1409bf4447 00007f1409c0cfe7 00007f1409c0e7de 00007f1409c18b49 00007f1409c19416 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4
0709 13:02:31.574004 (+536433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.574259 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308475776}
I20260709 13:02:31.601512 11516 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.646695  9330 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.601379 11471 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.647598  8726 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:31.646731  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.624007 11332 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:31.688352  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.694850  8732 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:31.694801  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:31.696072  8726 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:31.654820 11450 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:31.699676  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:31.748301  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.772102  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.757074  9330 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:31.728256  8729 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:31.707858  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.799376  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:31.801985  8724 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.779248  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:31.806777  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 101584us
W20260709 13:02:31.793826 11532 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 13:02:31.808660  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:31.810034  8658 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:31.794675  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:31.813597  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:31.741521 11514 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.614s	user 0.008s	sys 0.004s
I20260709 13:02:31.759507 11488 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.782238  8725 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:02:31.815312  8651 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:02:31.687295  8466 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 520005us
I20260709 13:02:31.818334  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:31.840673 11398 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:31.841549 11398 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.818362  8658 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:31.844507  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:31.845130  8658 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:31.846072  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:31.846210  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:31.847205  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:31.847293 11397 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:31.847853 11397 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.848184 11397 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.848304  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:31.849045  8658 raft_consensus.cc:2368] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7.
I20260709 13:02:31.850310  8659 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
W20260709 13:02:31.851648 11531 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 13:02:31.851770  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:31.852628  8659 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:31.691504 11212 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.818588  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:31.818848  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:31.820283  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:31.858052  8655 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.818608  8650 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260709 13:02:31.701165 11383 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.794305 11279 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.823155  8657 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:31.819680  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:31.866636  8653 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:31.822736  9323 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:31.837700  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:31.814605  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:31.867780  8730 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:31.855082  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:31.819376  9322 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:31.871124  8650 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:31.875869 11494 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:31.860675  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.860529  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:31.855324  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.861539 10816 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.822892 11450 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.687677 11409 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.876395  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:31.834992 11488 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:31.832232 11514 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.858728 11456 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:02:31.878885 11212 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:31.879215  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:31.880020  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 161922us
I20260709 13:02:31.880641  8651 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:31.697454 11332 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.882084  8658 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:31.884017  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:31.884470  8724 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:31.865567 11383 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:31.922559  9325 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:31.881057 11398 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:31.909178 11477 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:32.243804  8582 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:32.116379  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2216) took 1995 ms. Trace:
I20260709 13:02:32.250546  8655 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:32.246537  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.248584  8659 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:32.018232  8726 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:32.002939  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.914587 11455 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.258002  8724 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:31.912184 11377 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:32.267815  8724 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:32.250777  8652 rpcz_store.cc:276] 0709 13:02:30.121456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.142659 (+ 21203us) service_pool.cc:224] Handling call
0709 13:02:30.171064 (+ 28405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:30.171341 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:30.171375 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.726868 (+1555493us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b2c000b1fc8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:02:31.740946 (+ 14078us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.909972 (+169026us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:31.910009 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.910076 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.115997 (+205921us) raft_consensus.cc:1685] finished
0709 13:02:32.116026 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.116297 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6081792,"thread_start_us":133496,"threads_started":1,"wal-append.queue_time_us":337710}
I20260709 13:02:32.269289  8724 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:02:32.272055  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.941094 11456 consensus_queue.cc:1243] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:02:32.250906 11539 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.331s	user 0.001s	sys 0.000s
I20260709 13:02:31.979871 11450 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.980710 11276 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.256999 11477 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.248991 11332 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:32.248610 11500 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.942806 11516 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:32.298903  8696 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.691s (SENT)
I20260709 13:02:31.945116 11471 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:31.998056 11208 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:32.041688  9323 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:32.263625 11406 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.917037 10816 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:32.264261 11398 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.345296  8726 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:32.352344  8730 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:32.360777  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:32.361828  8582 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:32.294920 11401 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:32.302721 11477 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:32.308795 11488 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:32.306057 11279 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:32.305384 11383 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:32.349309  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 574) took 1193 ms. Trace:
I20260709 13:02:32.287494 11377 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:32.282992 11455 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:32.313647  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:32.371556 11540 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 13:02:32.453372  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:32.454396  8732 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:32.403033 11514 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:32.301887 11276 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:32.393947  9323 rpcz_store.cc:276] 0709 13:02:31.147089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.878834 (+731745us) service_pool.cc:224] Handling call
0709 13:02:32.340288 (+461454us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:32.457280  9323 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:02:32.458078  9323 raft_consensus.cc:606] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:02:32.458473  9323 raft_consensus.cc:614] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:32.458981  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39404 (request call id 0) took 1198 ms. Trace:
W20260709 13:02:32.459416 11528 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.876s	user 0.006s	sys 0.001s
I20260709 13:02:32.459343  9323 rpcz_store.cc:276] 0709 13:02:31.261150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.457063 (+1195913us) service_pool.cc:224] Handling call
0709 13:02:32.458872 (+  1809us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:32.461067  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 750) took 1723 ms. Trace:
I20260709 13:02:32.461611  9328 rpcz_store.cc:276] 0709 13:02:30.740392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.744058 (+  3666us) service_pool.cc:224] Handling call
0709 13:02:30.789528 (+ 45470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.852261 (+ 62733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.603001 (+750740us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b5400815954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:32.460834 (+857833us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b5400815950. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:32.461008 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":193168000}
W20260709 13:02:32.463043  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 890443us
I20260709 13:02:32.470554  8582 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:32.334340 11500 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:32.472254  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:32.338256 11406 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:32.420071  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.371922 11456 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.478631  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:32.478307 11456 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:32.479352  8575 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
W20260709 13:02:32.382198 11469 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.835s	user 0.011s	sys 0.000s
I20260709 13:02:32.373106 11401 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:32.342406 11398 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:32.439231  8696 leader_election.cc:336] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.831s (SENT)
I20260709 13:02:32.451508 10816 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:32.310662 11397 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:32.984123  8547 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.041s	sys 0.098s
I20260709 13:02:32.522797  9328 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:33.214664  8547 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.950s	user 0.050s	sys 0.122s
W20260709 13:02:33.214905  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.209784  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:32.583849  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 747) took 1850 ms. Trace:
I20260709 13:02:32.540185  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 568) took 1797 ms. Trace:
I20260709 13:02:32.479645  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:32.566592  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2313) took 1374 ms. Trace:
I20260709 13:02:33.215934  9329 rpcz_store.cc:276] 0709 13:02:30.735075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.738039 (+  2964us) service_pool.cc:224] Handling call
0709 13:02:30.738378 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.867942 (+129564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.583501 (+1715559us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54002779d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:32.583768 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":292450560}
I20260709 13:02:33.216881  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:32.482973  9327 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:02:33.205056  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2208) took 1611 ms. Trace:
I20260709 13:02:33.218531  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 765) took 2176 ms. Trace:
I20260709 13:02:33.219038  9327 rpcz_store.cc:276] 0709 13:02:31.056421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.102774 (+ 46353us) service_pool.cc:224] Handling call
0709 13:02:31.104161 (+  1387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:32.482638 (+1378477us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b5400214ba0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:02:33.218274 (+735636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.218488 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7403392}
I20260709 13:02:33.218396  8731 rpcz_store.cc:276] 0709 13:02:31.593191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.593300 (+   109us) service_pool.cc:224] Handling call
0709 13:02:31.593658 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.594018 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.594047 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.594466 (+   419us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.856715 (+262249us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:31.856755 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.856825 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.452787 (+595962us) raft_consensus.cc:1685] finished
0709 13:02:32.452808 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.204958 (+752150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3568,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":106,"thread_start_us":237250,"threads_started":1,"wal-append.queue_time_us":333236}
I20260709 13:02:32.480618  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:32.618535  8482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 202) took 11935 ms (client timeout 15000 ms). Trace:
I20260709 13:02:33.220643  8582 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:32.506673  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:32.760300  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.490010  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 406) took 2329 ms. Trace:
I20260709 13:02:32.547034  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 753) took 1426 ms. Trace:
I20260709 13:02:33.224025  9326 rpcz_store.cc:276] 0709 13:02:31.120769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.149372 (+ 28603us) service_pool.cc:224] Handling call
0709 13:02:32.546817 (+1397445us) spinlock_profiling.cc:243] Waited 63.6 ms on lock 0x7b5400517a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:32.546960 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146302208}
W20260709 13:02:33.209075  8623 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.005s	sys 0.005s
W20260709 13:02:33.225216  8623 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.005s	sys 0.005s
I20260709 13:02:32.482177  8578 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:33.225623  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 681562us
I20260709 13:02:33.216331  9325 rpcz_store.cc:276] 0709 13:02:30.742404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.772722 (+ 30318us) service_pool.cc:224] Handling call
0709 13:02:30.773102 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.245042 (+1471940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.539908 (+294866us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54004dec54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:32.540115 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2642056576}
I20260709 13:02:33.223066  8654 rpcz_store.cc:276] 0709 13:02:30.161646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.197381 (+ 35735us) service_pool.cc:224] Handling call
0709 13:02:30.225722 (+ 28341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.407220 (+1181498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.489661 (+1082441us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400410ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:32.489896 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2697682816}
I20260709 13:02:33.230597  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:32.395440 11507 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.971s	user 0.006s	sys 0.006s
I20260709 13:02:33.217183  8584 rpcz_store.cc:276] 0709 13:02:31.191692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.495853 (+304161us) service_pool.cc:224] Handling call
0709 13:02:31.496304 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.573969 (+ 77665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.566208 (+992239us) spinlock_profiling.cc:243] Waited 57.3 ms on lock 0x7b54000f91d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:32.566507 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131844480}
I20260709 13:02:33.233955  9329 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 13:02:33.234805  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 573) took 2088 ms. Trace:
I20260709 13:02:33.235158  9329 rpcz_store.cc:276] 0709 13:02:31.146796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.233329 (+2086533us) service_pool.cc:224] Handling call
0709 13:02:33.233704 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.234542 (+   838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.234749 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.239907  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:32.797663 11260 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:32.545982 11500 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:32.573170 11545 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.731s	user 0.008s	sys 0.022s
W20260709 13:02:32.610059 11537 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.768s	user 0.003s	sys 0.009s
W20260709 13:02:33.244064  8698 connection.cc:256] RPC call timeout handler was delayed by -1.44075366200002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20260709 13:02:33.255769  8698 connection.cc:256] RPC call timeout handler was delayed by -1.3713859999999727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
W20260709 13:02:32.572410 11533 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.796s	user 0.003s	sys 0.003s
W20260709 13:02:32.484871 11543 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.257s	user 0.000s	sys 0.000s
I20260709 13:02:32.806043  9322 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:32.590418 11484 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.978s	user 0.013s	sys 0.000s
I20260709 13:02:32.365357 11208 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.243445 11430 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:32.593420 11406 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.246786  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2199) took 2094 ms. Trace:
I20260709 13:02:33.266477  8727 rpcz_store.cc:276] 0709 13:02:31.152069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.152288 (+   219us) service_pool.cc:224] Handling call
0709 13:02:31.152597 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.160794 (+  8197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.160829 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.172448 (+ 11619us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.465649 (+293201us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:31.465699 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.465781 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.458769 (+992988us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:33.239613 (+780844us) raft_consensus.cc:1685] finished
0709 13:02:33.239635 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.246709 (+  7074us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2219,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":8890,"spinlock_wait_cycles":10986752,"thread_start_us":227815,"threads_started":1,"wal-append.queue_time_us":306517}
I20260709 13:02:33.249733  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:33.228360 11450 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.928s	user 0.006s	sys 0.006s
I20260709 13:02:32.900456  9323 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:33.250617  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 754) took 2121 ms. Trace:
I20260709 13:02:33.241920 11507 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:32.588080 11409 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.197345 11485 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.253504  8583 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:33.256606  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.265887  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 763) took 1693 ms. Trace:
I20260709 13:02:32.518154 11364 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.196118  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1869) took 1502 ms. Trace:
I20260709 13:02:32.543708  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:33.220781  8482 rpcz_store.cc:269] 0709 13:02:20.682781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.682932 (+   151us) service_pool.cc:224] Handling call
0709 13:02:21.710729 (+1027797us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b54002c0010. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af20b2 00007f140fa6ba89 00007f140fa03da6 00007f140f9d5088 00007f140fad337e 00007f140df942a4 00007f140df94232 00007f140df94161 00007f140df940dc
0709 13:02:32.618438 (+10907709us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:28.143519 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:28.144174 (+   655us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:28.144228 (+    54us) write_op.cc:435] Acquired schema lock
0709 13:02:28.144247 (+    19us) tablet.cc:662] Decoding operations
0709 13:02:28.145382 (+  1135us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:28.145493 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:28.145520 (+    27us) tablet.cc:685] Acquiring locks for 11 operations
0709 13:02:28.146663 (+  1143us) tablet.cc:701] Row locks acquired
0709 13:02:28.146709 (+    46us) write_op.cc:260] PREPARE: finished
0709 13:02:28.146899 (+   190us) write_op.cc:270] Start()
0709 13:02:28.147074 (+   175us) write_op.cc:276] Timestamp: P: 1783602148146831 usec, L: 0
0709 13:02:28.147097 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:02:28.147995 (+   898us) log.cc:844] Serialized 4301 byte log entry
0709 13:02:29.124946 (+976951us) op_driver.cc:464] REPLICATION: finished
0709 13:02:29.260500 (+135554us) write_op.cc:301] APPLY: starting
0709 13:02:29.290752 (+ 30252us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:29.292160 (+  1408us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:29.292331 (+   171us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:29.300764 (+  8433us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:29.301131 (+   367us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 13:02:29.301187 (+    56us) write_op.cc:312] APPLY: finished
0709 13:02:29.301630 (+   443us) log.cc:844] Serialized 83 byte log entry
0709 13:02:29.326567 (+ 24937us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:29.331167 (+  4600us) write_op.cc:454] Released schema lock
0709 13:02:29.342360 (+ 11193us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:29.378672 (+ 36312us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:29.378723 (+    51us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2184,"reported_tablets":20,"spinlock_wait_cycles":4186880,"child_traces":[["op",{"apply.queue_time_us":135275,"apply.run_cpu_time_us":4524,"apply.run_wall_time_us":111967,"num_ops":11,"prepare.queue_time_us":414603,"prepare.run_cpu_time_us":63584,"prepare.run_wall_time_us":167154,"raft.queue_time_us":227394,"raft.run_cpu_time_us":30925,"raft.run_wall_time_us":113872,"replication_time_us":977668,"thread_start_us":654181,"threads_started":4,"wal-append.queue_time_us":267926,"wal-append.run_cpu_time_us":41771,"wal-append.run_wall_time_us":1880848}]]}
I20260709 13:02:33.251124 11307 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.280424 11430 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 13:02:33.286087 11307 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.286507 11398 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.247866 11486 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:32.932333 11332 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.626s	user 0.008s	sys 0.002s
I20260709 13:02:33.235260 10816 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.238430 11276 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.238870 11377 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.294906 11377 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.295583 10816 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.298105 11307 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.298323 11276 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:32.508543 11534 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.733s	user 0.000s	sys 0.006s
I20260709 13:02:33.298879 11276 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.299847  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2319) took 1808 ms. Trace:
I20260709 13:02:33.301079 11276 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.265280  9327 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:33.265383 11466 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.269815 11552 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:33.270613 11477 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.887s	user 0.011s	sys 0.000s
I20260709 13:02:33.247136 11260 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.271030  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:33.233727 11434 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 13:02:33.278589  9328 rpcz_store.cc:276] 0709 13:02:31.121018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.462548 (+1341530us) service_pool.cc:224] Handling call
0709 13:02:32.522603 (+ 60055us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.214939 (+692336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.242436 (+ 27497us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b5400702bd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:33.242585 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137349504}
I20260709 13:02:32.585201 11447 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:33.261245 11469 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.201653 11494 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:33.277731 11551 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:33.289641  8729 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:02:33.291862 11486 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.328887 11477 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.295699 11377 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.238950 11455 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.300259  8579 rpcz_store.cc:276] 0709 13:02:31.491703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.529140 (+ 37437us) service_pool.cc:224] Handling call
0709 13:02:31.529687 (+   547us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.532402 (+  2715us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.532803 (+   401us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.534730 (+  1927us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.800640 (+265910us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:31.800701 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.800795 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.507934 (+707139us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:33.299540 (+791606us) raft_consensus.cc:1685] finished
0709 13:02:33.299564 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.299783 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1407,"raft.queue_time_us":2507,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":1801,"spinlock_wait_cycles":2831232,"thread_start_us":157589,"threads_started":1,"wal-append.queue_time_us":218072}
I20260709 13:02:33.240454 11440 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.281647 11456 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.348694 11440 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.293064 11332 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.328334  8578 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:33.326400 11466 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.285306  9322 rpcz_store.cc:276] 0709 13:02:31.572724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.289024 (+716300us) service_pool.cc:224] Handling call
0709 13:02:33.263653 (+974629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.295231 11507 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:33.333745  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 571) took 2513 ms (client timeout 2048 ms). Trace:
I20260709 13:02:33.302960 11364 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.341933  9330 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:33.298099 10816 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:33.346101 11249 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.346544 11560 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:33.347393  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 840631us
I20260709 13:02:33.347673 11562 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:33.327123  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 578) took 1745 ms. Trace:
I20260709 13:02:33.330103 11447 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:33.360392  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1870) took 1662 ms. Trace:
I20260709 13:02:33.385912  8729 rpcz_store.cc:276] 0709 13:02:31.698445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.839941 (+141496us) service_pool.cc:224] Handling call
0709 13:02:31.840308 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.334832 (+1494524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.360034 (+ 25202us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54000893d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:33.360325 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3332165376}
I20260709 13:02:33.364679 11260 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.367193 11397 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.371496  8577 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:33.328568 11552 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.350668 11558 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:33.375224  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 757) took 2253 ms. Trace:
I20260709 13:02:33.395569  9330 rpcz_store.cc:276] 0709 13:02:31.121759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.521915 (+1400156us) service_pool.cc:224] Handling call
0709 13:02:32.522340 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.365128 (+842788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.374989 (+  9861us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400319054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:33.375162 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1884152704}
I20260709 13:02:33.337389 11469 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.378927  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.382246  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.382795  8724 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:02:33.373844  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 934512us
I20260709 13:02:33.398532  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.384680 11486 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.384819  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.404036  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.404852  8730 raft_consensus.cc:2410] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 13:02:33.341782  8480 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 4, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 4 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:33.384258 11562 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.387187 11377 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.366931 11401 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:33.363451  9324 rpcz_store.cc:269] 0709 13:02:30.812561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.813805 (+  1244us) service_pool.cc:224] Handling call
0709 13:02:32.543466 (+1729661us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:33.326087 (+782621us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5954432}
I20260709 13:02:33.393772  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:33.420984  8728 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.376467 11208 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:33.398640  8731 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:33.383195  9327 rpcz_store.cc:276] 0709 13:02:31.573662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.250302 (+1676640us) service_pool.cc:224] Handling call
0709 13:02:33.262435 (+ 12133us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.318552 (+ 56117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.318789 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.399224  8729 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:33.406371 11447 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.406936 11440 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.406831  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.408159 11466 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.408965  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:33.426451  9330 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:33.368034 11364 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.432065  8724 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:33.432452  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1745) took 2027 ms. Trace:
I20260709 13:02:33.409821  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.433697  8732 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:33.373678 11507 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 13:02:33.390769  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:33.418849  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 752) took 2289 ms. Trace:
I20260709 13:02:33.421578  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.450016  9330 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:33.419593  8480 catalog_manager.cc:5697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 4, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 4 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:33.383178 11249 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.427529  8731 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:33.428965  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:33.430287 11552 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:33.428965  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.475059  8733 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:33.436432  8724 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:33.385497 11560 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.443893  8732 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:33.480266  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:33.444411  8729 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:02:33.432816  8580 rpcz_store.cc:276] 0709 13:02:31.405508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.517533 (+112025us) service_pool.cc:224] Handling call
0709 13:02:31.524979 (+  7446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.525339 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.525372 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.534640 (+  9268us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.803638 (+268998us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:31.803691 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.803785 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.569274 (+765489us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b240011e748. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:02:33.430552 (+861278us) spinlock_profiling.cc:243] Waited 6.24 ms on lock 0x7b240011e748. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b67fb0 00007f14097de488 00007f14097debe8 00007f14097b6398 00007f14097b176a 00007f14097b0bb6
0709 13:02:33.432231 (+  1679us) raft_consensus.cc:1685] finished
0709 13:02:33.432250 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.432399 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1942,"raft.queue_time_us":874,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":87,"spinlock_wait_cycles":44476416,"thread_start_us":217008,"threads_started":1,"wal-append.queue_time_us":294412}
W20260709 13:02:33.419695  8696 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.870s (SENT)
I20260709 13:02:33.483472  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:33.453059  8728 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:33.420200 11562 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.441076 11561 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:33.457329 11249 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.458050  9328 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:33.424896 11558 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:33.488587 11558 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.458309 11469 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:33.491982  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:33.459115 11555 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.212s	user 0.003s	sys 0.000s
I20260709 13:02:33.459972 11447 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:33.461537 11397 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.424605 11332 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.462986  8726 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:33.473780  9324 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:33.478408  8480 catalog_manager.cc:5697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 3, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:33.487932  8718 tablet_service.cc:1558] Processing DeleteTablet for tablet 9e8a9411169f407cba1b44443edc9840 with delete_type TABLET_DATA_TOMBSTONED (TS 1dc41db601fe4b91aa6d461b504da0e7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40398
I20260709 13:02:33.287719  8725 rpcz_store.cc:276] 0709 13:02:31.693378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.839927 (+146549us) service_pool.cc:224] Handling call
0709 13:02:31.840346 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.840707 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.840740 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.841338 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.997993 (+156655us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:32.015569 (+ 17576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.228710 (+213141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.576559 (+347849us) raft_consensus.cc:1685] finished
0709 13:02:32.576597 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.196019 (+619422us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3901,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":49408,"thread_start_us":99108,"threads_started":1,"wal-append.queue_time_us":575962}
I20260709 13:02:33.499042  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:33.492610  9322 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 5: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:33.495410 11208 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:33.514804  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:33.516407  9330 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:33.533798  9327 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:02:33.502651 11552 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 13:02:33.498275 11249 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.501801 11477 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:33.536791 11456 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:33.449618  9323 rpcz_store.cc:276] 0709 13:02:31.120491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.460269 (+1339778us) service_pool.cc:224] Handling call
0709 13:02:32.460613 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.278195 (+817582us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.393472 (+115277us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54008b4054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:33.410053 (+ 16581us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":417620736}
I20260709 13:02:33.549082  9327 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:33.554509  9328 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:33.514091 10816 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:33.566973  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 82299.6us
I20260709 13:02:33.542810  8480 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 4, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 4 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:02:33.575840  9297 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:33.579694  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:33.503382 11397 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.617015  8578 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
W20260709 13:02:33.618160  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49969.3us
I20260709 13:02:33.495069 11561 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.621270  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2226) took 1776 ms. Trace:
I20260709 13:02:33.625240  8650 rpcz_store.cc:276] 0709 13:02:31.845121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.871947 (+ 26826us) service_pool.cc:224] Handling call
0709 13:02:31.911363 (+ 39416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.934572 (+ 23209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.934630 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.973082 (+ 38452us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.404460 (+431378us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:32.404512 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.404586 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.393707 (+989121us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:33.550422 (+156715us) raft_consensus.cc:1685] finished
0709 13:02:33.550443 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.621200 (+ 70757us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4155,"raft.queue_time_us":6156,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":140,"spinlock_wait_cycles":2773248,"thread_start_us":148349,"threads_started":1,"wal-append.queue_time_us":1205007}
I20260709 13:02:33.536947 11494 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 13:02:33.555985 11558 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.551254 11456 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.545220 11208 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:33.617157 11552 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:33.629887  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 390011us
I20260709 13:02:33.614884 10816 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.622488  8480 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 3, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:33.629705 11332 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.546509 11249 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.645524  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:33.645885 11469 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:33.646185 11466 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:33.629609 11563 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 13:02:33.646520 11208 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.650826 11397 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.655943  9331 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:33.665678  8578 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:33.657321  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:33.633258 11566 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.690482  8575 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260709 13:02:33.691308  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:33.692436 11397 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.693567  8577 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:33.693691 11551 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:33.693883 11397 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.693066 11377 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:33.694226 11484 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:33.695379 11260 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:33.675885 11500 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.695860 11398 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:33.696614 11398 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:33.696388 11500 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.697532  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:33.692454  8583 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:33.697463 11398 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.698475  8625 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:33.698689 11398 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.698993 11500 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.689847 11486 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:33.700659 11486 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:33.695852 11260 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.701679 11260 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:33.701771 11377 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:33.639951 11552 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.702231 11434 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:33.702490 11377 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:33.702796 11552 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:33.703001 11377 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.704635 11434 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:33.705065 11434 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:33.691001  8575 pending_rounds.cc:85] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 4
I20260709 13:02:33.658787 11558 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:33.693610  9331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 567) took 2951 ms (client timeout 1520 ms). Trace:
W20260709 13:02:33.708607  9331 rpcz_store.cc:269] 0709 13:02:30.742009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.742129 (+   120us) service_pool.cc:224] Handling call
0709 13:02:33.693388 (+2951259us) spinlock_profiling.cc:243] Waited 2.86 s on lock 0x7b5400552654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:33.693545 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6584214272}
I20260709 13:02:33.701589 11486 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.669497  8578 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260709 13:02:33.640439 11456 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:33.711176 11570 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:33.708302  8575 pending_rounds.cc:107] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.5
I20260709 13:02:33.662467 11560 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.657534 11565 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:02:33.658394 11332 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.712634 11434 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.713699 11567 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:33.658896 10816 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:33.715377  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:33.701953 11307 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:33.718366 11406 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:33.669658 11450 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.657285 11401 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20260709 13:02:33.719513 11397 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.658288 11364 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:33.726931  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 760) took 2155 ms. Trace:
I20260709 13:02:33.722725  8575 raft_consensus.cc:2917] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Aborting config change with OpId 2.5: Aborted: Op aborted by new leader
I20260709 13:02:33.750010  8575 raft_consensus.cc:2889] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:33.715091  8578 pending_rounds.cc:85] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 5
I20260709 13:02:33.750823  8578 pending_rounds.cc:107] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260709 13:02:33.751123  8578 raft_consensus.cc:2889] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:33.669068  8480 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:02:33.730619  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:33.758239  9325 rpcz_store.cc:276] 0709 13:02:31.571914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.237458 (+1665544us) service_pool.cc:224] Handling call
0709 13:02:33.237685 (+   227us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.237817 (+   132us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:33.624921 (+387104us) log.cc:844] Serialized 14 byte log entry
0709 13:02:33.726445 (+101524us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.726505 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:33.726677 (+   172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.726752 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.726873 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":205,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":3559,"thread_start_us":467781,"threads_started":2}
I20260709 13:02:33.768146  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:33.772531  8726 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:33.783661 11577 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:33.785351 11577 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:33.786175 11577 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:33.805795  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 75267.2us
W20260709 13:02:33.788098  9329 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.010s	sys 0.015s
W20260709 13:02:33.809131  9329 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.010s	sys 0.015s
I20260709 13:02:33.810346  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 576) took 2237 ms. Trace:
I20260709 13:02:33.811213  9329 rpcz_store.cc:276] 0709 13:02:31.573081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.240494 (+1667413us) service_pool.cc:224] Handling call
0709 13:02:33.240793 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.241119 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.241153 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.241644 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 13:02:33.809867 (+568223us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:33.809927 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.810027 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.810087 (+    60us) raft_consensus.cc:1685] finished
0709 13:02:33.810104 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.810273 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36677,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2954,"thread_start_us":545977,"threads_started":1,"wal-append.queue_time_us":533392}
W20260709 13:02:33.797076  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.006s	sys 0.007s
W20260709 13:02:33.813265  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.007s	sys 0.007s
I20260709 13:02:33.814131  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:33.798049  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.008s	sys 0.006s
I20260709 13:02:33.731803 11556 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:33.815212 11556 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:33.815788 11556 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:33.801498  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 448202us
I20260709 13:02:33.787312 11579 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:33.729146 11567 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:33.795195  8624 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.000s	sys 0.014s
W20260709 13:02:33.823773  8624 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.000s	sys 0.014s
W20260709 13:02:33.824536  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:33.796823  9326 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.010s	sys 0.004s
W20260709 13:02:33.825389  9326 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.010s	sys 0.004s
I20260709 13:02:33.826185  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 759) took 2599 ms. Trace:
I20260709 13:02:33.826567  9326 rpcz_store.cc:276] 0709 13:02:31.227205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.236473 (+2009268us) service_pool.cc:224] Handling call
0709 13:02:33.236717 (+   244us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.236953 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.236971 (+    18us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.237315 (+   344us) log.cc:844] Serialized 26 byte log entry
0709 13:02:33.825818 (+588503us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:33.825863 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.825936 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.825978 (+    42us) raft_consensus.cc:1685] finished
0709 13:02:33.825990 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.826128 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":560057,"threads_started":1,"wal-append.queue_time_us":523476}
W20260709 13:02:33.797610  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.006s	sys 0.008s
W20260709 13:02:33.828342  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.006s	sys 0.008s
W20260709 13:02:33.797843  8622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.004s	sys 0.009s
W20260709 13:02:33.821080  9296 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.826s (SENT) [suppressed 27 similar messages]
W20260709 13:02:33.828827  8622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.004s	sys 0.009s
W20260709 13:02:33.821457  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.008s	sys 0.006s
I20260709 13:02:33.733994 11307 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:33.829804  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 554982us
I20260709 13:02:33.831658  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:33.832953  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:33.834515  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 589158us
W20260709 13:02:33.835876  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 416140us
I20260709 13:02:33.836563  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:33.801714 11434 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:33.806183 11577 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.732599 11276 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.090s
I20260709 13:02:33.840900  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:33.836606  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.949s (SENT)
I20260709 13:02:33.841327 11455 raft_consensus.cc:2764] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:33.844074 11578 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:33.844921 11578 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:33.845474 11578 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:33.846475 11578 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.850739 11582 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.850622 11260 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.707836 11566 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:33.733736 11450 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.820554 11477 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:33.843565  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:33.836534 11469 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.860322  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:33.861222  9326 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 13:02:33.888091  9326 pending_rounds.cc:85] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 6
I20260709 13:02:33.888473  9326 pending_rounds.cc:107] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:02:33.888790  9326 raft_consensus.cc:2889] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:33.861557  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:33.861891  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:33.873157  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.889711  9325 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:33.889762  9329 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.868177  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.890969  8726 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:33.869971  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.870600  8724 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 13:02:33.861954  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47943.7us
I20260709 13:02:33.834223 11567 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:33.900218  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 68660us
I20260709 13:02:33.839341 11249 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:33.883618  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41978 (request call id 83) took 29068 ms (client timeout 29999 ms). Trace:
I20260709 13:02:33.841650 11484 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.903753  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:33.908914  9329 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:33.913470  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.906138  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:33.859777 11581 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.935410  8732 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 13:02:33.915859  8575 rpcz_store.cc:269] 0709 13:02:04.773530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.773686 (+   156us) service_pool.cc:224] Handling call
0709 13:02:04.849044 (+ 75358us) log.cc:844] Serialized 487 byte log entry
0709 13:02:05.152196 (+303152us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b2c000014c8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851ea0a 00007f140851e693 00007f140851e631 00007f1409c52b52 00007f1409c52cf5 00007f1409b8f691 00007f1409b6feaa 00007f1409b76d53 00007f1409b76cd9 00007f1409b76c69
0709 13:02:05.206968 (+ 54772us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b18003d0e40. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8f142 00007f1409b6feaa 00007f1409b76d53
0709 13:02:05.545133 (+338165us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b54001741a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8f142 00007f1409b6feaa 00007f1409b76d53 00007f1409b76cd9 00007f1409b76c69 00007f1409b76c31 00007f1409b75dfd 000000000038f354 000000000038f199 00007f1406b29475
0709 13:02:33.838377 (+28293244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":122720,"raft.queue_time_us":122106,"raft.run_cpu_time_us":6982,"raft.run_wall_time_us":549613,"spinlock_wait_cycles":43323776,"thread_start_us":105603,"threads_started":1,"wal-append.queue_time_us":106049,"wal-append.run_cpu_time_us":2475,"wal-append.run_wall_time_us":1033550}
W20260709 13:02:33.899101 11409 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.617s	user 0.009s	sys 0.003s
I20260709 13:02:33.940770  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:33.944041  9331 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:33.852058 11582 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:33.941352  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 86324.4us
I20260709 13:02:33.898559 11551 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260709 13:02:33.956574  9325 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:33.922602 11561 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.945356 11584 tablet_replica.cc:333] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:02:33.898307 11377 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.835279 11579 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.962857  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:33.903807 11469 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.949258 11455 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:33.949380 11559 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.969722  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.969740  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:33.926631 11587 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260709 13:02:33.930955 11484 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:33.976946  8732 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:33.981765  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:33.982019  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:33.982602  8729 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:33.984704  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.986096  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.987246  8724 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:33.988539  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.990195  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:33.992137  8724 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:33.955613  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:33.959625 11447 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.983337  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:34.006090  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.008104 11484 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:33.969081 11589 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 13:02:34.009719 11589 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:34.010557 11589 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:33.975332 11455 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.011847 11455 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:33.994513 11559 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:34.007467  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 138419us
I20260709 13:02:34.002301  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:34.013252  8731 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:33.979554 11276 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:34.014143 11276 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:34.014292  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:34.014555 11276 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:34.014994  8731 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 3.
I20260709 13:02:34.015480 11276 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:34.016243  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:34.017194  8731 raft_consensus.cc:2410] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:33.994180 11582 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:33.941527 11307 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:34.040485  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:34.045292  8724 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:33.976958  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 140358us
W20260709 13:02:34.047775  8696 leader_election.cc:336] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.018s (SENT)
I20260709 13:02:34.048213  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:33.969149  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:34.005867 11561 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:33.945966 11581 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:34.060340  8733 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:34.066886  8733 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:34.059213  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 155538us
I20260709 13:02:34.075519  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:34.081779  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:34.082098  8729 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:34.085309  8729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bf63362bb9b4409299528319b7472b04"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:02:34.085958  8729 raft_consensus.cc:606] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:34.094986  8729 raft_consensus.cc:614] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:34.067747  8733 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:34.021361 11332 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:34.050993 11570 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:34.021807 11593 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:34.117602  8733 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.995674 11406 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 13:02:34.019438 11447 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:34.005204 11573 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.134s
I20260709 13:02:34.078648 11588 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:34.140564  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 187542us
I20260709 13:02:34.122104 11260 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:34.146123 11260 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:34.149694 11364 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:34.150163 11364 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:34.151291 11593 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:34.152397 11593 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:34.153661 11406 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:34.160939  8733 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 13:02:34.150981 11364 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:34.166806  8733 pending_rounds.cc:85] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 2
I20260709 13:02:34.203821  8733 pending_rounds.cc:107] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 13:02:34.204169  8733 raft_consensus.cc:2889] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:34.200042  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:34.208593  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 162525us
W20260709 13:02:34.213192  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 71830.6us
I20260709 13:02:34.140169 11588 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:34.204913 11540 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:02:34.203429 11409 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.222795  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:34.127306 11332 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260709 13:02:34.208544 11590 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:34.220651 11588 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:34.221905 11450 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:34.229295 11561 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:34.278540  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:34.339325  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60864.4us
W20260709 13:02:34.340662  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 63788.2us
W20260709 13:02:34.330636  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.334856  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:34.278424  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:34.351897  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:34.364251  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:34.368623  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:34.368594  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:34.373658  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:34.241837 11409 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:34.378854  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:34.280031 11561 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:34.340476 11440 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:34.377884  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:34.347831 11377 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:34.381256  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:34.382118  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:34.382931  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:34.406306  8583 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:34.406704  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:34.383033 11469 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:34.377233  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:34.366621 11332 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:34.412043  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:34.398488  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 316717us
I20260709 13:02:34.277835 11604 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:34.354172 11398 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:34.407395  9331 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:34.407758  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:34.408375  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:34.363873 11599 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 13:02:34.382655 11566 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:34.400388  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Aborted: Op aborted by new leader
W20260709 13:02:34.431915  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 356401us
I20260709 13:02:34.434528  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:34.399771 11561 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:34.428566  9328 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:34.419909  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:34.383625 11581 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:34.409327 11598 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 13:02:34.476487  8622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.008s	sys 0.015s
W20260709 13:02:34.478591  8622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.008s	sys 0.015s
W20260709 13:02:34.479996  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 619677us
I20260709 13:02:34.410671 11605 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:34.413559 11600 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:34.432861 11398 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:34.475162 11477 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:02:34.470819 11585 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.416s	user 0.001s	sys 0.000s
I20260709 13:02:34.419483 11332 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:34.485864  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.011s	sys 0.014s
I20260709 13:02:34.416291 11584 raft_consensus.cc:2243] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Raft consensus shutting down.
W20260709 13:02:34.489631  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55204.9us
W20260709 13:02:34.490159  8730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.011s	sys 0.020s
I20260709 13:02:34.484102 11581 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 13:02:34.489768 11602 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:34.508587  8730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.011s	sys 0.021s
W20260709 13:02:34.490756  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70908.2us
I20260709 13:02:34.472198  9324 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:02:34.494463  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.002s	sys 0.009s
I20260709 13:02:34.414088 11594 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:34.498970  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 449551us
I20260709 13:02:34.499992 11447 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:34.501667  8466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 133488us
I20260709 13:02:34.522590 11602 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:02:34.505116  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.011s	sys 0.014s
I20260709 13:02:34.483235 11579 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:34.495541 11605 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.508884  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:34.472795 11601 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
I20260709 13:02:34.525985 11604 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:34.499393 11398 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:34.503595 11477 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:34.530488 11552 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.826s	user 0.012s	sys 0.000s
I20260709 13:02:34.512531 11430 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 13:02:34.538100  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:34.515261  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.002s	sys 0.010s
W20260709 13:02:34.602151 11567 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.679s	user 0.006s	sys 0.014s
I20260709 13:02:34.603904 11567 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:34.542058  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 584505us
I20260709 13:02:34.606531  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.607693  8650 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:34.500902 11600 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:34.558496  8725 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:02:34.559638  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 582) took 1168 ms. Trace:
W20260709 13:02:34.584702  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 306306us
W20260709 13:02:34.583009  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 69086us
W20260709 13:02:34.583926  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 59791.9us
I20260709 13:02:34.535861 11579 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.159s
I20260709 13:02:34.525673 11594 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:34.617569  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:34.793824  8653 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:34.620388  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.622982  8655 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:34.626145  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.630429  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.609102  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.794474  8730 rpcz_store.cc:276] 0709 13:02:33.391560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.462807 (+ 71247us) service_pool.cc:224] Handling call
0709 13:02:33.476346 (+ 13539us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.476685 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.476710 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.477222 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 13:02:33.797450 (+320228us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:34.513925 (+716475us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.559138 (+ 45213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.559264 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.559337 (+    73us) raft_consensus.cc:1685] finished
0709 13:02:34.559359 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.559552 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":343142,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":24480,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":1586,"thread_start_us":1027549,"threads_started":3,"wal-append.queue_time_us":304526}
I20260709 13:02:34.619243  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.610206  8656 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:02:34.800137  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.793864  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:34.631306  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.801021  8650 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:34.634665  8657 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 13:02:34.738534  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 356965us
W20260709 13:02:34.696384  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 90427.7us
I20260709 13:02:34.807372  8652 raft_consensus.cc:2410] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:34.531785 11618 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:34.809154  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:34.810205  8658 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:34.810148  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.804952  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:34.538324 11605 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:34.771311 11591 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.362s	user 0.001s	sys 0.000s
W20260709 13:02:34.887427  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.011s	sys 0.021s
W20260709 13:02:34.887826  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.011s	sys 0.021s
I20260709 13:02:34.563490 11477 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:34.844750  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:34.853070  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48182.6us
I20260709 13:02:34.537751 11332 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:34.747344 11602 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:34.774382 11556 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.958s	user 0.000s	sys 0.014s
I20260709 13:02:34.810413  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:34.745082 11584 raft_consensus.cc:2272] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Raft consensus is shut down!
W20260709 13:02:34.888118  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 547698us
I20260709 13:02:34.730034 11377 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.807489 11409 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:34.927284  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 133474us
W20260709 13:02:34.929840  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 391778us
W20260709 13:02:34.953902  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 143569us
I20260709 13:02:34.954618  9297 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:34.921146  8658 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:34.954073  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 151787us
I20260709 13:02:34.911993  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.915511  9331 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:34.893759  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.956346  8653 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:34.893872  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.924031  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.899075  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:34.897672  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.964699  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.744211 11440 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:34.909572  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.741797 11307 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.563119 11621 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:34.995532  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:34.692464 11552 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:34.995859  8655 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:34.997581  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:35.017073 11456 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.299s	user 0.006s	sys 0.004s
I20260709 13:02:34.741937 11578 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.241s
I20260709 13:02:34.997023  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.029211  8653 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:34.876621 11581 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.126s
W20260709 13:02:34.966686 11450 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.021s	sys 0.016s
I20260709 13:02:35.017441 11569 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.993119 11617 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.205s
I20260709 13:02:34.993371 11608 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:35.017771 11610 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.481s	user 0.000s	sys 0.001s
I20260709 13:02:34.997068 11551 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.018082  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.018347 11627 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:34.993534 11632 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:35.063318  8659 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:34.993113 11622 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 13:02:34.996204  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.030937  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:35.004616  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:35.042594 11450 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.022s	sys 0.017s
I20260709 13:02:35.000346  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:34.993376 11613 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:35.057106  8656 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:35.108621  8651 pending_rounds.cc:85] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 5
I20260709 13:02:35.058449  8652 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 13:02:35.109558  8652 pending_rounds.cc:85] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 2
W20260709 13:02:35.109364  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104877us
I20260709 13:02:35.110002  8652 pending_rounds.cc:107] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:02:35.110378  8652 raft_consensus.cc:2889] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:35.110580  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:35.059433  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:35.012244 11332 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:34.972728 11567 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:35.062033 11577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.000s	sys 0.025s
W20260709 13:02:35.040124 10816 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.306s	user 0.011s	sys 0.003s
I20260709 13:02:35.068948 11632 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.999806 11409 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.108s
I20260709 13:02:35.043797 11456 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.127377  9326 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:35.009745  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 216228us
I20260709 13:02:34.966169 11556 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:34.993435 11635 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
I20260709 13:02:35.116158  8651 pending_rounds.cc:107] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20260709 13:02:35.115378  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 157588us
I20260709 13:02:35.006688 11620 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:35.006706 11625 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.969936 11377 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:35.056015 11608 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:35.071354 11551 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:35.115638 11565 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:35.130263  8658 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:35.131048  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:35.136623  8651 raft_consensus.cc:2889] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:35.106091 11626 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.150s	user 0.002s	sys 0.000s
I20260709 13:02:35.123576  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:35.137197  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:35.002570 11307 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.126384  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1891) took 1254 ms. Trace:
I20260709 13:02:35.067867 11639 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:35.060973 11569 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.120142  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:35.117002 11613 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:35.133087 11607 raft_consensus.cc:670] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 13:02:35.231258  8729 rpcz_store.cc:276] 0709 13:02:33.872223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.253168 (+380945us) service_pool.cc:224] Handling call
0709 13:02:34.338164 (+ 84996us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.338601 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.338670 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.353563 (+ 14893us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.547165 (+193602us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:34.547248 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.547455 (+   207us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.126125 (+578670us) raft_consensus.cc:1685] finished
0709 13:02:35.126149 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.126318 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":75332,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":57,"thread_start_us":143226,"threads_started":1,"wal-append.queue_time_us":220822}
I20260709 13:02:35.239450  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:35.233875  8726 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:35.235798  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98690.9us
W20260709 13:02:35.231568  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 234020us
I20260709 13:02:35.244457  8653 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:02:35.245055  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.245805  8658 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:35.125600 11577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.000s	sys 0.026s
I20260709 13:02:35.119359 11638 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:35.052819  8655 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:35.251087  8659 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:35.121899 11627 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:35.238198  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 118169us
W20260709 13:02:35.297147  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53638.8us
I20260709 13:02:35.126592 11623 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:35.271070  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 316465us
I20260709 13:02:35.336561  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:35.337854  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:35.323329  8650 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:35.151412 11610 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:02:35.227927 11646 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:35.138182 11620 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.309051  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.354240  8655 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:35.243292 11611 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.372s	user 0.001s	sys 0.001s
I20260709 13:02:35.150718 11565 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.226579 11556 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:35.312480  8657 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:35.318677  8652 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:35.319134  8698 connection.cc:256] RPC call timeout handler was delayed by -1.3112713809999832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16110000000000002s for a response
I20260709 13:02:35.367718  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:35.367841  8698 connection.cc:256] RPC call timeout handler was delayed by -1.2113713809999922s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20260709 13:02:35.368623  8698 connection.cc:256] RPC call timeout handler was delayed by -1.2014713809999762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
I20260709 13:02:35.368710  8729 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:35.304560  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:35.146442 11608 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:35.239077 11409 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:35.374729  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1885) took 1504 ms. Trace:
I20260709 13:02:35.375495  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.377470  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:35.355901  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 360032us
I20260709 13:02:35.250002 11616 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.128s
I20260709 13:02:35.228391 11456 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:35.233196 11614 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.375s	user 0.000s	sys 0.009s
I20260709 13:02:35.251794 11579 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:35.304996 11639 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.133548 11637 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.186s
I20260709 13:02:35.125998 11632 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:35.298786 11642 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.299646 11447 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:35.298795 11638 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:35.136170 11434 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.283s	user 0.013s	sys 0.001s
W20260709 13:02:35.706468 11398 consensus_meta.cc:287] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.015s	user 0.012s	sys 0.000s
I20260709 13:02:35.707098 11398 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:35.364233  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 233242us
I20260709 13:02:35.711246  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:35.369179  8698 connection.cc:256] RPC call timeout handler was delayed by -1.1438713809999967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
W20260709 13:02:35.711967  8698 connection.cc:256] RPC call timeout handler was delayed by -1.0178713810000204s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
I20260709 13:02:35.320186  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.125988 10816 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.375171  8727 rpcz_store.cc:276] 0709 13:02:33.870033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.103710 (+233677us) service_pool.cc:224] Handling call
0709 13:02:34.104095 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.117153 (+ 13058us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.117188 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:34.117228 (+    40us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:34.117260 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.374494 (+1257234us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b54004727d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:35.374538 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.374665 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29542144}
I20260709 13:02:35.248479 11613 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:35.681864  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2231) took 1811 ms. Trace:
I20260709 13:02:35.725158  8724 rpcz_store.cc:276] 0709 13:02:33.869966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.045995 (+176029us) service_pool.cc:224] Handling call
0709 13:02:34.046293 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.046619 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:34.252915 (+206296us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.476578 (+223663us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.482800 (+  6222us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.482980 (+   180us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:34.483021 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.483099 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.681534 (+1198435us) raft_consensus.cc:1685] finished
0709 13:02:35.681566 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.681764 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103175,"prepare.queue_time_us":106383,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":206848,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":30410,"thread_start_us":282667,"threads_started":2,"wal-append.queue_time_us":235319}
I20260709 13:02:35.727299  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:35.649684  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:35.547751  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 243233us
I20260709 13:02:35.363338 11623 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.644371 11584 ts_tablet_manager.cc:1916] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:02:35.553135 11552 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:35.475317 11657 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:35.649937 11650 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:35.370455 11600 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:35.366345 11565 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:35.455947  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 60026.7us
W20260709 13:02:35.488312 11588 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.135s	user 0.012s	sys 0.000s
I20260709 13:02:35.469836  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:35.785876  8729 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:35.456879 11636 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:35.311550 11578 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 13:02:35.476003  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.358355 11594 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:35.363848 11620 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:35.146759 11625 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:35.320994 11607 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 13:02:35.456481 11631 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:35.311290 11581 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:02:35.482383  8655 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:35.230744 11440 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.357549 11646 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:35.475142 11633 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.428s	user 0.000s	sys 0.001s
W20260709 13:02:35.657485  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.093s	sys 0.170s
W20260709 13:02:35.813908  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.093s	sys 0.174s
W20260709 13:02:35.814507  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 819273us
I20260709 13:02:35.640138 11660 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:35.476949 11629 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.330s	user 0.000s	sys 0.000s
I20260709 13:02:35.691829  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:35.816932  9329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 772) took 1472 ms (client timeout 1590 ms). Trace:
W20260709 13:02:35.817334  9329 rpcz_store.cc:269] 0709 13:02:34.344597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.347253 (+  2656us) service_pool.cc:224] Handling call
0709 13:02:35.691627 (+1344374us) spinlock_profiling.cc:243] Waited 9.07 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:35.816871 (+125244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20859648}
I20260709 13:02:35.818471  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:35.695060 11628 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:35.696578  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1889) took 1825 ms. Trace:
I20260709 13:02:35.719774 11642 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:35.702589  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2232) took 1487 ms. Trace:
W20260709 13:02:35.848004  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 135072us
I20260709 13:02:35.702802 11551 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:35.703305  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:35.724380  8650 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:35.552875 11409 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.138s
W20260709 13:02:35.375001 11562 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.874s	user 0.014s	sys 0.000s
I20260709 13:02:35.850185  8732 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:35.754683  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:35.850769  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 591) took 1979 ms (client timeout 1837 ms). Trace:
I20260709 13:02:35.690740  9324 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:35.758071  9327 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:35.726763 11260 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:35.739264 11276 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.723s	user 0.010s	sys 0.003s
I20260709 13:02:35.719035 11638 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:35.726950 11579 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:35.737149 11627 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:35.738180 11601 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:35.760452 11624 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:02:35.753273 11401 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 13:02:35.790740 11588 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.762890  9323 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:35.797127 11634 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:02:35.797827 11661 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.765527 11657 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.776454 11650 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.778066 11552 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:35.806443  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:35.894639 11552 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.895545  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:35.896795  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:35.897028 11552 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:35.811815  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 81845.6us
I20260709 13:02:35.819065  8733 rpcz_store.cc:276] 0709 13:02:33.871608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.205436 (+333828us) service_pool.cc:224] Handling call
0709 13:02:34.205837 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.219504 (+ 13667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.219549 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.220184 (+   635us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.474645 (+254461us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:34.474698 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.474772 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.696274 (+1221502us) raft_consensus.cc:1685] finished
0709 13:02:35.696302 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.696485 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7065,"raft.queue_time_us":198355,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":2447,"thread_start_us":218882,"threads_started":1,"wal-append.queue_time_us":503285}
I20260709 13:02:35.698150 11649 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.815578  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:35.800556 11398 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:35.804026 11569 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:35.761968 11652 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20260709 13:02:35.851087  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 96486us
W20260709 13:02:35.851114  8732 rpcz_store.cc:269] 0709 13:02:33.871761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.980973 (+109212us) service_pool.cc:224] Handling call
0709 13:02:35.703154 (+1722181us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:35.850699 (+147545us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35666816}
I20260709 13:02:35.807497 11607 consensus_queue.cc:1243] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:35.816125 11660 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.854590  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:35.813774 11631 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:35.857791  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 497949us
W20260709 13:02:35.948064  8549 connection.cc:256] RPC call timeout handler was delayed by -1.1267961249999985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25920000000000004s for a response
I20260709 13:02:35.870138 11208 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
W20260709 13:02:35.856281 11561 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.373s	user 0.014s	sys 0.000s
I20260709 13:02:35.868068  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.966598  8652 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:35.907675 11650 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:35.869701  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.855540 10816 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:35.864745 11455 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.852s	user 0.003s	sys 0.012s
I20260709 13:02:35.970520  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1890) took 2098 ms. Trace:
I20260709 13:02:35.852185 11434 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
W20260709 13:02:35.879297  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 582) took 1602 ms (client timeout 1716 ms). Trace:
I20260709 13:02:35.888870 11579 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.899004  9329 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:35.897341  8729 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:35.855706 11594 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
W20260709 13:02:35.926968  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 111438us
I20260709 13:02:35.997711  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 422) took 1328 ms. Trace:
I20260709 13:02:35.996920  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:35.853621 11562 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.936482  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.033876  8583 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:35.936720  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:35.937691  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:35.995528  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 429) took 1117 ms. Trace:
I20260709 13:02:35.943967  8655 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:35.848145  8725 rpcz_store.cc:276] 0709 13:02:34.215480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.797376 (+581896us) service_pool.cc:224] Handling call
0709 13:02:34.797735 (+   359us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:35.013316 (+215581us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.013363 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:35.026444 (+ 13081us) log.cc:844] Serialized 52 byte log entry
0709 13:02:35.309148 (+282704us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:35.309207 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.702198 (+392991us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54001e69d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:35.702248 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.702293 (+    45us) raft_consensus.cc:1685] finished
0709 13:02:35.702307 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.702518 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3262,"raft.queue_time_us":15655,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":22133,"spinlock_wait_cycles":487067904,"thread_start_us":181590,"threads_started":1,"wal-append.queue_time_us":575856}
I20260709 13:02:36.035887  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:36.032280  8651 rpcz_store.cc:276] 0709 13:02:34.668925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.084781 (+415856us) service_pool.cc:224] Handling call
0709 13:02:35.085300 (+   519us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:35.400282 (+314982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.400351 (+    69us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:35.401088 (+   737us) log.cc:844] Serialized 52 byte log entry
0709 13:02:35.808644 (+407556us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:35.808715 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.808808 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.997494 (+188686us) raft_consensus.cc:1685] finished
0709 13:02:35.997513 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.997651 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76891,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":270785,"threads_started":1,"wal-append.queue_time_us":429690}
I20260709 13:02:35.946027  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:35.915659 11477 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.948s	user 0.014s	sys 0.000s
I20260709 13:02:35.862830 11276 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:35.935986 11605 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.019s	user 0.009s	sys 0.003s
I20260709 13:02:36.038236  8584 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:35.885717 11657 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:36.044953  8731 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:36.046041 11562 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:35.962105  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:36.050102  8733 raft_consensus.cc:2410] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:36.051142  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:36.051776  8733 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:36.052609  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:35.943118 11629 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 13:02:36.057946  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 774) took 1683 ms. Trace:
I20260709 13:02:35.796038  9331 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:35.968717  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:36.059360  8581 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:36.058305  9330 rpcz_store.cc:276] 0709 13:02:34.374949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.375083 (+   134us) service_pool.cc:224] Handling call
0709 13:02:34.406945 (+ 31862us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.407284 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.407312 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.431369 (+ 24057us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.729276 (+297907us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:34.955130 (+225854us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.973996 (+ 18866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.974520 (+   524us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.057731 (+1083211us) raft_consensus.cc:1685] finished
0709 13:02:36.057754 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.057889 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14247,"prepare.queue_time_us":290191,"raft.queue_time_us":521212,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":51961,"spinlock_wait_cycles":16093568,"thread_start_us":366977,"threads_started":2,"wal-append.queue_time_us":522917}
I20260709 13:02:35.860635 11260 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.899338 11642 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:35.971129  8728 rpcz_store.cc:276] 0709 13:02:33.871948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.212876 (+340928us) service_pool.cc:224] Handling call
0709 13:02:34.213209 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.245206 (+ 31997us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:34.364783 (+119577us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.800897 (+436114us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.810950 (+ 10053us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.811103 (+   153us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:34.811136 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.811196 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.125918 (+314722us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b180012b8c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b29522 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:35.650205 (+524287us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b180012b8c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:35.970286 (+320081us) raft_consensus.cc:1685] finished
0709 13:02:35.970306 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.970430 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":155565,"prepare.queue_time_us":73632,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":4373,"raft.queue_time_us":444364,"raft.run_cpu_time_us":10431,"raft.run_wall_time_us":74079,"spinlock_wait_cycles":41337472,"thread_start_us":133552,"threads_started":1,"wal-append.queue_time_us":407199}
I20260709 13:02:36.068326  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:35.939255 11668 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:35.975226 11625 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:35.948737 11486 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:35.977982  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:35.944756 11649 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.924957 11646 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.998695  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:36.089763  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 91173.2us
I20260709 13:02:36.042061 11594 consensus_queue.cc:1243] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:35.938048  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:36.034992  8575 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:36.037066  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:35.819975 11628 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:36.037942  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.041119  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.039287  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.038317  8579 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:36.052983 11276 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:35.881278 11627 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.053530 11605 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.054512 11620 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.995075 11623 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.055187 11636 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:36.055560 11581 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:36.040565 11634 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 13:02:36.056434 11477 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:35.998109 11455 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.066234  8729 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:35.973656 11208 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:35.886186 11661 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.034020  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.259438  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.081928  8728 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:36.101414  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.260344  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.260779  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1900) took 1298 ms. Trace:
I20260709 13:02:36.077709 11486 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:35.947453 11660 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:36.072255 11668 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.040556 11570 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:36.266957  9324 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 3.
I20260709 13:02:36.271169 11570 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.272079  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:36.040669 11567 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 13:02:36.037529  8657 rpcz_store.cc:276] 0709 13:02:34.878066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.901883 (+ 23817us) service_pool.cc:224] Handling call
0709 13:02:35.995330 (+1093447us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540026c354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:35.995462 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":730215808}
I20260709 13:02:36.094882  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1896) took 1133 ms. Trace:
I20260709 13:02:36.081357  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.261224  8728 rpcz_store.cc:276] 0709 13:02:34.962632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.068303 (+1105671us) service_pool.cc:224] Handling call
0709 13:02:36.068588 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.260506 (+191918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.260724 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:36.275900  8654 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:36.276120  9331 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 4: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:36.276618  8724 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:36.260584 11208 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:36.108166  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.262630  8584 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:36.262931  9330 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:36.282951  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.284585  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.262137  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:36.148065  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:36.222815  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:36.259439  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:36.092036  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.037381  9323 rpcz_store.cc:269] 0709 13:02:34.277971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.337215 (+ 59244us) service_pool.cc:224] Handling call
0709 13:02:35.649531 (+1312316us) spinlock_profiling.cc:243] Waited 9.01 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:35.876382 (+226851us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20728192}
I20260709 13:02:36.267994  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.287186  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.262558 11634 consensus_queue.cc:1243] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:36.081229 11569 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.971626 11561 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:36.149834 11608 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.606s	user 0.010s	sys 0.002s
I20260709 13:02:36.176767 11649 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.273741  8727 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:36.148211 11562 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:36.147013  8581 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:36.267256 11581 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.263639  8725 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:36.178221 11260 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.279404  8733 rpcz_store.cc:276] 0709 13:02:34.961535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.061279 (+1099744us) service_pool.cc:224] Handling call
0709 13:02:36.094178 (+ 32899us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.094417 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.094456 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:36.094511 (+    55us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:36.094571 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.094648 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.094802 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:36.287843  9323 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:36.271199 11486 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.289688  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.290382 11570 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.291956 11455 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:36.081482 11619 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.485s	user 0.010s	sys 0.006s
W20260709 13:02:36.329212 11619 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.733s	user 0.013s	sys 0.013s
W20260709 13:02:36.293740  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71025us
I20260709 13:02:36.330446  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:36.296499  8724 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:36.317090 11665 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.369s	user 0.001s	sys 0.001s
I20260709 13:02:36.297493  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:36.298561  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:36.301582  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 263779us
I20260709 13:02:36.302928  8575 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:36.307396  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.307432  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:36.311210  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:36.346256  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.347458  8583 raft_consensus.cc:2410] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:36.348503  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.349537  8575 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:36.350728  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.351920  8575 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:36.275928 11276 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.314364  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.184574 11657 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.278378  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.273918 11668 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:36.293030 11588 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.281647 11638 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.359630  8583 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:02:36.361685  8583 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:36.320771  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:36.320935  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:36.306648 10816 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:36.324756  9331 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:36.269873  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.333813  8727 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:36.363086  8584 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:36.334810 11671 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:36.337550 11486 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:36.339265  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:36.083431 11642 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.193s
I20260709 13:02:36.316520 11562 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:36.353335  8657 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:36.375077  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.376129  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:36.376895 11565 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.577s	user 0.008s	sys 0.000s
I20260709 13:02:36.377910 11593 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.378242 11455 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:36.378499 11593 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.379098  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 775) took 3005 ms. Trace:
I20260709 13:02:36.379499  9327 rpcz_store.cc:276] 0709 13:02:33.373849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.552026 (+178177us) service_pool.cc:224] Handling call
0709 13:02:33.618701 (+ 66675us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.854134 (+2235433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.378853 (+524719us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54006c3754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:36.379033 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4918880640}
W20260709 13:02:36.380414  8650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 431) took 1468 ms (client timeout 1752 ms). Trace:
I20260709 13:02:36.380625 11593 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.377107  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.381228 11593 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.382305 11477 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.382727 11661 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:36.357267  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49912.8us
I20260709 13:02:36.386885 11646 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:36.357973 11675 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:36.387699 11638 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.387853 11675 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:36.388876 11675 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.320562  8725 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:36.392472 11657 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:36.359661  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62265.7us
W20260709 13:02:36.380878  8650 rpcz_store.cc:269] 0709 13:02:34.911955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.995398 (+ 83443us) service_pool.cc:224] Handling call
0709 13:02:36.380252 (+1384854us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400424854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:36.380353 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":592345472}
I20260709 13:02:36.362481  8581 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:36.363057  8579 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:36.331889  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:36.397604  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:36.398080  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66270.2us
I20260709 13:02:36.373307  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.400444  9331 raft_consensus.cc:2410] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:36.373622  9323 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:36.374159 11562 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:36.315129 11608 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.385758  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.372583 11620 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.401585  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:36.398276  8584 raft_consensus.cc:2376] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7. Re-sending same reply.
I20260709 13:02:36.380751  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:36.414538  9327 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:36.354102 11652 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:36.408078  8583 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:02:36.313488 11569 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.396458  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:36.414206  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:36.417606  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.432890  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.433763  8656 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:36.429030  8579 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 13:02:36.429692  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.398447 11594 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:36.427295  8581 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:36.444222  8575 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 13:02:36.397801 11631 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.409561 11638 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:36.451110  8732 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:36.451495  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:36.452278  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:36.452471  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:36.453053  9331 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:36.313289 11628 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.454427  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:36.456876  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60472.9us
I20260709 13:02:36.457660  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.462705  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:36.464054  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.421952 11652 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.369149 11276 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.377327 11565 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.382325 11593 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.468116  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.469869  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:36.470618  9331 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:36.345418 11208 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:36.455117 11332 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.327s	user 0.008s	sys 0.005s
I20260709 13:02:36.412358 11570 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:02:36.450886 10816 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:36.453683 11600 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:36.467370 11632 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.713s	user 0.011s	sys 0.000s
I20260709 13:02:36.455381 11638 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:36.505069  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.506181  8584 raft_consensus.cc:2410] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:36.502960  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.503772  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.468109 11671 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:36.519438  9331 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:36.507261  8656 raft_consensus.cc:2376] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7. Re-sending same reply.
I20260709 13:02:36.441614 11678 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 13:02:36.547722  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.472786 11661 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:36.498473 11579 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:36.527356  8729 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:36.505163 11434 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:36.567809  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.510588 11208 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:36.472826 11652 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:36.573041  8656 raft_consensus.cc:2410] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:36.725274  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.726816  8650 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:36.726464  8657 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:36.727097  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.727809  8656 raft_consensus.cc:2376] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7. Re-sending same reply.
I20260709 13:02:36.728088  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:36.722370  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 259595us
I20260709 13:02:36.729104  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.722182  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.730582  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2243) took 1007 ms. Trace:
I20260709 13:02:36.732932  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:36.733913  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 340427us
I20260709 13:02:36.733148  8727 rpcz_store.cc:276] 0709 13:02:35.723256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.363650 (+640394us) service_pool.cc:224] Handling call
0709 13:02:36.364065 (+   415us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:36.403734 (+ 39669us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.403763 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:36.410787 (+  7024us) log.cc:844] Serialized 52 byte log entry
0709 13:02:36.581782 (+170995us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:36.581838 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.581923 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.729727 (+147804us) raft_consensus.cc:1685] finished
0709 13:02:36.729779 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.730221 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3334,"raft.queue_time_us":39308,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":35842,"thread_start_us":114414,"threads_started":1,"wal-append.queue_time_us":189453}
I20260709 13:02:36.737586  9324 raft_consensus.cc:2410] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:02:36.739666  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:36.588209 11434 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:36.613135 11469 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:36.740942 11434 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:36.642143 11276 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.741107 11469 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.742522 11469 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:36.744952  8581 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:36.559275 10816 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:36.613364 11559 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.658648 11569 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.674347 11562 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.747534 10816 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.623509 11561 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.634599 11565 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.617026 11579 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:36.526186 11600 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:36.675629 11613 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.913s	user 0.009s	sys 0.004s
I20260709 13:02:36.753716 11579 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:36.526777 11601 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.613927 11581 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.646963 11584 ts_tablet_manager.cc:1929] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260709 13:02:36.630769 11593 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.626230 11608 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.575436 11627 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.546537 11628 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:36.510905 11632 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.511904 11638 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:36.748364 11569 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:36.501857 11650 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.561846 11654 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 13:02:36.626583 11660 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.605249 11661 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:36.866027 11661 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.678732 11668 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.868904  8658 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:36.572959 11671 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:36.622941 11681 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:02:36.565927 11686 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:36.881835 11686 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.882444 11686 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:36.883801  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2233) took 1162 ms. Trace:
W20260709 13:02:36.644655 11684 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 13:02:36.614857 11687 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 13:02:36.885733 11601 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.754516 11579 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:36.770097 11652 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.774674 11584 log.cc:1199] T 9e8a9411169f407cba1b44443edc9840 P 1dc41db601fe4b91aa6d461b504da0e7: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/wals/9e8a9411169f407cba1b44443edc9840
I20260709 13:02:36.794996 11469 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.891279 11584 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40398 (request call id 80) took 3487 ms. Trace:
I20260709 13:02:36.891934 11469 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.884338  8728 rpcz_store.cc:276] 0709 13:02:35.721388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.276056 (+554668us) service_pool.cc:224] Handling call
0709 13:02:36.276387 (+   331us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:36.276861 (+   474us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.276896 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:36.295790 (+ 18894us) log.cc:844] Serialized 52 byte log entry
0709 13:02:36.441419 (+145629us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:36.441476 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.442714 (+  1238us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.883201 (+440487us) raft_consensus.cc:1685] finished
0709 13:02:36.883245 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.883555 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":849,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":17653,"thread_start_us":121039,"threads_started":1,"wal-append.queue_time_us":152329}
W20260709 13:02:36.807638  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71773.8us
I20260709 13:02:36.796207 11569 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.843205 11581 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:36.848250  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.848702  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.849668 11650 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.850319 11600 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.764554 11613 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:36.853257  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.854100  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 399531us
I20260709 13:02:36.858505  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:36.863032 11654 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:36.869364  9330 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:36.876701 11660 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.876861 11668 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.876688  8581 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:36.858300 11562 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.880017 11671 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.884927  8651 raft_consensus.cc:2368] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5.
I20260709 13:02:36.894675  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:36.889957 11652 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.895399 11569 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:36.868922  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 425) took 2089 ms (client timeout 1843 ms). Trace:
I20260709 13:02:36.903864  9323 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:36.906783  8656 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:36.908815  8581 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:02:36.909359 11613 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.915153  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.899856 11654 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:36.904460  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:36.921917 11602 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:36.915249 11434 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.925341 11652 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:36.925971 11652 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.926471 11652 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:36.927424  8650 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:36.899811 11581 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.932296 11705 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:36.934435 11579 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.935726  8727 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:36.936367 11632 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.937460  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:36.923916 11687 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:36.924719  8656 raft_consensus.cc:2376] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5. Re-sending same reply.
W20260709 13:02:36.906023  8654 rpcz_store.cc:269] 0709 13:02:34.779998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.799993 (+ 19995us) service_pool.cc:224] Handling call
0709 13:02:36.868055 (+2068062us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540018ecd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:36.868675 (+   620us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2945301376}
I20260709 13:02:36.939385 11332 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:36.916393  9323 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:02:36.940954 11332 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:36.941579  9323 raft_consensus.cc:606] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:36.942102  9323 raft_consensus.cc:614] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:36.942361 11332 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:36.943164  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:36.943928  9323 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:36.940042 11687 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:36.946594 11687 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.942811  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.948040 11434 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.947322 11601 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:36.949704 11638 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.891794 11584 rpcz_store.cc:276] 0709 13:02:33.437908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.480881 (+ 42973us) service_pool.cc:224] Handling call
0709 13:02:33.884071 (+403190us) ts_tablet_manager.cc:1151] Deleting tablet 9e8a9411169f407cba1b44443edc9840
0709 13:02:33.897752 (+ 13681us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 13:02:36.330062 (+2432310us) tablet_metadata.cc:665] Metadata flushed
0709 13:02:36.646786 (+316724us) tablet_metadata.cc:665] Metadata flushed
0709 13:02:36.891102 (+244316us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":118156,"tablet-delete.queue_time_us":309355,"thread_start_us":258194,"threads_started":1}
I20260709 13:02:36.955217  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:36.957062 11581 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.962913  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 574) took 2584 ms. Trace:
I20260709 13:02:36.958842 11638 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.963883  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:36.963426  8580 rpcz_store.cc:276] 0709 13:02:34.377890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.378020 (+   130us) service_pool.cc:224] Handling call
0709 13:02:34.384346 (+  6326us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:34.393196 (+  8850us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.393240 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:34.417001 (+ 23761us) log.cc:844] Serialized 52 byte log entry
0709 13:02:36.848031 (+2431030us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b1800476260. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:02:36.919537 (+ 71506us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:36.920470 (+   933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.920657 (+   187us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.939304 (+ 18647us) raft_consensus.cc:1685] finished
0709 13:02:36.939315 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.962503 (+ 23188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8306,"raft.queue_time_us":91155,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":5970,"spinlock_wait_cycles":20573824,"thread_start_us":29826,"threads_started":1,"wal-append.queue_time_us":2456639}
I20260709 13:02:36.965763  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:36.966935 11654 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.968511  8728 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:02:36.969447  9322 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:36.969590 11704 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:02:36.970216  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 585) took 2088 ms. Trace:
I20260709 13:02:36.970685  9322 rpcz_store.cc:276] 0709 13:02:34.882191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.888671 (+  6480us) service_pool.cc:224] Handling call
0709 13:02:36.967077 (+2078406us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:36.967320 (+   243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.969909 (+  2589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.970129 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8554752}
I20260709 13:02:36.972472  8728 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:36.973068 11579 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:36.973330  9325 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 13:02:36.973924  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44071.2us
I20260709 13:02:36.975378  8728 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:36.975809  8728 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:36.975993  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:36.976765  8728 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:36.978381  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:36.982930  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 588) took 1189 ms. Trace:
I20260709 13:02:36.974117  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 583) took 2694 ms. Trace:
I20260709 13:02:36.948616 11687 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:36.969026 11654 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:36.984722  9325 rpcz_store.cc:276] 0709 13:02:34.279260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.345471 (+ 66211us) service_pool.cc:224] Handling call
0709 13:02:36.972534 (+2627063us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:36.972865 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:36.973820 (+   955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.974041 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24753920}
I20260709 13:02:36.969053  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 783) took 1004 ms. Trace:
I20260709 13:02:36.983538  9329 rpcz_store.cc:276] 0709 13:02:35.793226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.818230 (+ 25004us) service_pool.cc:224] Handling call
0709 13:02:36.982287 (+1164057us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b5400519354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:36.982601 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184087424}
I20260709 13:02:36.981200  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 779) took 1647 ms. Trace:
W20260709 13:02:36.989566  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 72742.3us
I20260709 13:02:36.987782  9331 rpcz_store.cc:276] 0709 13:02:35.964942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.556583 (+591641us) service_pool.cc:224] Handling call
0709 13:02:36.556978 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:36.751710 (+194732us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.751755 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:36.752549 (+   794us) log.cc:844] Serialized 26 byte log entry
0709 13:02:36.769096 (+ 16547us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:36.769141 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.807174 (+ 38033us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.968557 (+161383us) raft_consensus.cc:1685] finished
0709 13:02:36.968578 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.968831 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2211,"raft.queue_time_us":10097,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":6629,"thread_start_us":207341,"threads_started":2,"wal-append.queue_time_us":186537}
I20260709 13:02:37.001446  9322 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:37.006085 11692 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 13:02:37.006610  9329 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:37.008203  9329 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:37.012133 11307 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.013273  9327 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:37.014103  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 786) took 1035 ms. Trace:
I20260709 13:02:37.014457  9327 rpcz_store.cc:276] 0709 13:02:35.978921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.976758 (+997837us) service_pool.cc:224] Handling call
0709 13:02:36.977196 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.013806 (+ 36610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.014021 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:36.997624 11704 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:36.998324 11581 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:37.016985  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.018343  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:37.020172  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.022354  9325 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:37.022516  8580 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:37.023228  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 787) took 1025 ms. Trace:
I20260709 13:02:37.023676  9325 rpcz_store.cc:276] 0709 13:02:35.998167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.997335 (+999168us) service_pool.cc:224] Handling call
0709 13:02:37.020203 (+ 22868us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.022968 (+  2765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.023171 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.022534  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:37.029425  8581 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:37.032521 11377 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:36.988783  9328 rpcz_store.cc:276] 0709 13:02:35.334012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.334180 (+   168us) service_pool.cc:224] Handling call
0709 13:02:35.366924 (+ 32744us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.537974 (+171050us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.538010 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:35.558228 (+ 20218us) log.cc:844] Serialized 26 byte log entry
0709 13:02:35.819896 (+261668us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:35.969867 (+149971us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.970291 (+   424us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.980204 (+1009913us) spinlock_profiling.cc:243] Waited 95.4 ms on lock 0x7b54008b4054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:36.980309 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.980433 (+   124us) raft_consensus.cc:1685] finished
0709 13:02:36.980611 (+   178us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.980945 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75490,"prepare.queue_time_us":233548,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":221649,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":27470,"spinlock_wait_cycles":219414784,"thread_start_us":286669,"threads_started":2,"wal-append.queue_time_us":262278}
W20260709 13:02:37.036249  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.036790 11364 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:37.036558 11569 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:37.037451  8696 connection.cc:256] RPC call timeout handler was delayed by -1.96249371600004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17420000000000002s for a response
W20260709 13:02:37.038446  8696 connection.cc:256] RPC call timeout handler was delayed by -1.5610937160000162s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21880000000000002s for a response
W20260709 13:02:37.038782  8696 connection.cc:256] RPC call timeout handler was delayed by -1.5106937159999916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22440000000000004s for a response
I20260709 13:02:37.039817 11613 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:37.040273  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:37.040601 11613 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.041608 11562 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:37.042449 11562 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.043291 11613 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.044862 11562 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:37.045573 11628 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:37.046232 11628 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.047794  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:37.048727  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70583.8us
W20260709 13:02:37.037132  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.055546 11579 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:37.056249 11579 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:37.056778 11579 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:37.059062  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:37.061474  9327 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:37.063549 11562 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:37.063700 11398 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:37.064225 11562 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:37.068761  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.073418  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:02:37.074209  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.071770 11704 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:37.072376 11569 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:37.075822 11704 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.076965  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:37.078217 11377 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:37.064353 11398 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.082338 11594 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.082979 11594 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.084396 11581 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:37.084882 11578 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:37.085821 11578 consensus_queue.cc:1243] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:02:37.086361 11377 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:37.086794 11692 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:37.087743 11692 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.088860 11692 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.090377 11692 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:37.091600 11702 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 13:02:37.095433 11377 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:37.096203 11377 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.098500 11377 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.102747  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:37.103232 11398 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:37.103996 11398 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:37.104391 11398 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.104743 11398 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:37.105954 11648 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.689s	user 0.007s	sys 0.005s
I20260709 13:02:37.086486 11578 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:02:37.107124 11578 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:37.107559 11578 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.108801 11578 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.110101 11578 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:37.112437  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.113473  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 747005us
W20260709 13:02:37.113986  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 112335us
I20260709 13:02:37.078536 11704 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.125787  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.094074 11661 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:37.129621  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:37.130537  8576 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:37.129984 11661 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.093019 11562 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.142650  8582 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:37.143446  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 578) took 1197 ms. Trace:
I20260709 13:02:37.143769  8582 rpcz_store.cc:276] 0709 13:02:35.945919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.946081 (+   162us) service_pool.cc:224] Handling call
0709 13:02:37.141763 (+1195682us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:37.142200 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.143167 (+   967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.143374 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5243776}
I20260709 13:02:37.144654 11639 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.094301 11628 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:37.145815  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.147219  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.147243 11632 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:37.147941  8654 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.148403 11632 consensus_queue.cc:1243] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:02:37.149044 11632 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:37.149595 11632 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:37.151059 11632 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.152853 11632 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.155045  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.155786  8652 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:37.156895  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.157827  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:37.158496  9328 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:37.159570  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:37.160734  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:37.162931 11686 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.172143  9328 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:37.169836  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2362) took 1311 ms. Trace:
I20260709 13:02:37.165387  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2244) took 2497 ms. Trace:
I20260709 13:02:37.173727  8658 rpcz_store.cc:276] 0709 13:02:35.858119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.858656 (+   537us) service_pool.cc:224] Handling call
0709 13:02:36.868318 (+1009662us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:36.876042 (+  7724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.169494 (+293452us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540018ecd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.169696 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2320544640}
I20260709 13:02:37.162953  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.174759  8659 rpcz_store.cc:276] 0709 13:02:34.667731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.065303 (+397572us) service_pool.cc:224] Handling call
0709 13:02:35.123217 (+ 57914us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.296087 (+172870us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.164826 (+1868739us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400001754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.165168 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":292631808}
W20260709 13:02:37.179373  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 102533us
I20260709 13:02:37.152796  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:37.183364  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.183677  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2365) took 1322 ms. Trace:
I20260709 13:02:37.186605  8656 raft_consensus.cc:2410] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:37.140043 11652 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.186774  8655 rpcz_store.cc:276] 0709 13:02:35.861293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.162460 (+1301167us) service_pool.cc:224] Handling call
0709 13:02:37.183592 (+ 21132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.159930  9323 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:37.197867  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.194696  9325 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:37.204967  8576 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:37.191191 11377 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.203159  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.210110  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 579) took 1259 ms. Trace:
I20260709 13:02:37.221498  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.206687  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.230336  8652 raft_consensus.cc:2410] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:37.220906  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2237) took 1498 ms. Trace:
I20260709 13:02:37.219493  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1884) took 3349 ms. Trace:
I20260709 13:02:37.234145  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:37.210532  8658 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:37.231124  8725 rpcz_store.cc:276] 0709 13:02:35.722331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.319199 (+596868us) service_pool.cc:224] Handling call
0709 13:02:36.319430 (+   231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.390679 (+ 71249us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.220291 (+829612us) spinlock_profiling.cc:243] Waited 940 us on lock 0x7b5400203ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.220640 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2161920}
I20260709 13:02:37.192006 11686 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:37.223806  8576 rpcz_store.cc:276] 0709 13:02:35.946211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.946323 (+   112us) service_pool.cc:224] Handling call
0709 13:02:37.129083 (+1182760us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:37.205577 (+ 76494us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4827392}
I20260709 13:02:37.239131  8659 raft_consensus.cc:2368] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:02:37.232287  8731 rpcz_store.cc:276] 0709 13:02:33.869707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.018863 (+149156us) service_pool.cc:224] Handling call
0709 13:02:34.020213 (+  1350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.045416 (+2025203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.218800 (+1173384us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b5400416754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.219227 (+   427us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4655526784}
I20260709 13:02:37.222287 11687 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.228883 11562 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:37.258589  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 55419.8us
I20260709 13:02:37.248405 11364 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.270814 11364 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.271278  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.271905  8652 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.274022  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.248795 11628 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:37.266283 11638 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:37.275110 11638 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.274960 11628 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.276535 11398 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.266247 11696 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:37.269666 11562 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.274200  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.275586 11638 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.269017  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 789) took 1237 ms. Trace:
I20260709 13:02:37.266410 11687 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.281288  8658 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:37.289907  8659 raft_consensus.cc:2410] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:37.296787  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 439) took 1032 ms. Trace:
I20260709 13:02:37.297277  8659 rpcz_store.cc:276] 0709 13:02:36.264673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.264787 (+1000114us) service_pool.cc:224] Handling call
0709 13:02:37.296698 (+ 31911us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.298815  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.285948  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.300176  8725 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:37.301177  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.302395  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.302901  8725 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 5
W20260709 13:02:37.306807  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 154317us
I20260709 13:02:37.286746  9331 rpcz_store.cc:276] 0709 13:02:36.031360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.019846 (+988486us) service_pool.cc:224] Handling call
0709 13:02:37.020349 (+   503us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.027592 (+  7243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:37.027632 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:37.028288 (+   656us) log.cc:844] Serialized 26 byte log entry
0709 13:02:37.067077 (+ 38789us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:37.067124 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.067216 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:37.268758 (+201542us) raft_consensus.cc:1685] finished
0709 13:02:37.268778 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.268954 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2005,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":187,"thread_start_us":38291,"threads_started":1,"wal-append.queue_time_us":160631}
I20260709 13:02:37.273205 11364 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:37.313928  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 143288us
I20260709 13:02:37.314122  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.319694 11562 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.319842 11608 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:37.319270 11696 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.316256  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:37.260978 11686 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:37.321116 11608 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:37.332299  8731 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:37.333740  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 140099us
I20260709 13:02:37.331672  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2371) took 1418 ms. Trace:
I20260709 13:02:37.334941  8659 rpcz_store.cc:276] 0709 13:02:35.903011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.305692 (+1402681us) service_pool.cc:224] Handling call
0709 13:02:37.321480 (+ 15788us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.331985 11608 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:37.337543 11430 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:02:37.338032 11608 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.339637 11608 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:37.341070 11661 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:37.342180  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:37.342934  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 58326.9us
I20260709 13:02:37.343688  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:37.319319 11398 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:37.347098 11398 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:37.347885 11398 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:37.347941  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:37.348683  8582 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.349594 11704 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:37.350355 11704 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:37.351854  8576 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:37.352440  8725 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:37.347020  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.354657 11687 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.355190 11562 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:37.355822  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:37.355861 11562 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.358660 11562 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.359261 11692 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:37.360780  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:37.361522  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:37.361706 11650 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:37.331856 11713 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 13:02:37.363195  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.340827 11556 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260709 13:02:37.362165 11605 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.378894  8654 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:37.337983 11652 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:37.383837  8582 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:37.385725  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.387667  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:37.386852  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.391062  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.374905 11650 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:37.402165  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.406093  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:37.402362  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:37.409925  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 50887.4us
I20260709 13:02:37.395258 11556 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:37.411936  8654 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:37.407214  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.375077 11687 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260709 13:02:37.414409  8659 raft_consensus.cc:2410] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:37.404594 11578 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:37.411424 11556 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:37.418989  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.410476 11686 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:37.404670 11650 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:37.429823 11632 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:37.430626 11687 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:37.431206 11687 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.383031 11661 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 13:02:37.437594 11705 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:37.437858 11617 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:37.439122 11705 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:37.439287 11620 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:37.439638 11705 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.439857 11578 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.439970 11620 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.440923 11578 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.441640 11620 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.383839 11579 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.382977 11692 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 13:02:37.445464  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.455466  8725 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:37.349207 11398 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.463797  8652 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
W20260709 13:02:37.499605  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 86449.4us
I20260709 13:02:37.501559  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 792) took 1279 ms. Trace:
I20260709 13:02:37.421208 11660 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:37.503683  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.453176 11692 consensus_queue.cc:1243] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:37.443445 11620 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.509208  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.504581  8731 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:37.504658  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.507314  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.419026 11556 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:37.514907  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 154177us
I20260709 13:02:37.515425 11556 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.515995  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:37.517004 11661 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:37.518426  8652 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [3.3-4.4]
I20260709 13:02:37.440771 11705 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.444475 11696 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:37.522938 11638 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.524263 11660 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:37.525247  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2240) took 1802 ms. Trace:
I20260709 13:02:37.535375  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2262) took 1459 ms. Trace:
I20260709 13:02:37.526404 11632 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:02:37.528333  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.537050  8659 rpcz_store.cc:276] 0709 13:02:36.076356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.463929 (+1387573us) service_pool.cc:224] Handling call
0709 13:02:37.535298 (+ 71369us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.528254  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.502010  9329 rpcz_store.cc:276] 0709 13:02:36.221952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.008968 (+787016us) service_pool.cc:224] Handling call
0709 13:02:37.009368 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.034887 (+ 25519us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:37.034919 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:37.035587 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 13:02:37.054424 (+ 18837us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:37.054451 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.054553 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:37.501287 (+446734us) raft_consensus.cc:1685] finished
0709 13:02:37.501310 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.501498 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":185,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":158,"thread_start_us":18373,"threads_started":1,"wal-append.queue_time_us":45500}
I20260709 13:02:37.432827 11687 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.530282  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2254) took 1454 ms. Trace:
W20260709 13:02:37.533169  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.517154 11628 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.542403  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2247) took 1469 ms. Trace:
I20260709 13:02:37.540359  8725 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:37.542804  8654 rpcz_store.cc:276] 0709 13:02:36.072635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.415791 (+1343156us) service_pool.cc:224] Handling call
0709 13:02:37.542327 (+126536us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.539454  8652 rpcz_store.cc:276] 0709 13:02:36.074407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.468329 (+1393922us) service_pool.cc:224] Handling call
0709 13:02:37.509366 (+ 41037us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:37.528958 (+ 19592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.529204 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.547559 11638 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:37.548631 11687 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.554127 11705 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:37.558228 11660 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.561501  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41978 (request call id 85) took 2964 ms. Trace:
W20260709 13:02:37.562373  8548 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.575s (SENT)
I20260709 13:02:37.562114  8577 rpcz_store.cc:276] 0709 13:02:34.597145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.597389 (+   244us) service_pool.cc:224] Handling call
0709 13:02:37.560801 (+2963412us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400179b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1c735 00007f1409c149fa 00007f140ebdc4c9 00007f1408158c54 00007f1408158be2 00007f1408158b11 00007f1408158a8c 00007f1408157d42 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.561213 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3405848960}
I20260709 13:02:37.563017  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:37.542982 11661 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.565707 11605 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:37.566821  8659 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:37.567899  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2255) took 1493 ms. Trace:
I20260709 13:02:37.568509  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:37.536770  8733 rpcz_store.cc:276] 0709 13:02:35.722802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.342434 (+619632us) service_pool.cc:224] Handling call
0709 13:02:36.342711 (+   277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:36.343044 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.343078 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:36.370405 (+ 27327us) log.cc:844] Serialized 26 byte log entry
0709 13:02:37.222534 (+852129us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b18002abc80. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:02:37.418145 (+195611us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:37.418184 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.418263 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:37.524724 (+106461us) raft_consensus.cc:1685] finished
0709 13:02:37.524749 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.525057 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1989,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":16622,"spinlock_wait_cycles":36038016,"thread_start_us":169998,"threads_started":1,"wal-append.queue_time_us":1123541}
I20260709 13:02:37.573225  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1774) took 1172 ms. Trace:
I20260709 13:02:37.573779  8584 rpcz_store.cc:276] 0709 13:02:36.401203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.429441 (+ 28238us) service_pool.cc:224] Handling call
0709 13:02:37.572781 (+1143340us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b540030c354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.573050 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153849088}
I20260709 13:02:37.575311  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1768) took 1544 ms. Trace:
I20260709 13:02:37.576503  8575 rpcz_store.cc:276] 0709 13:02:36.031069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.365845 (+334776us) service_pool.cc:224] Handling call
0709 13:02:36.443985 (+ 78140us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:36.444701 (+   716us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.574947 (+1130246us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b540017a550. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.575147 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178724096}
I20260709 13:02:37.578020  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.578279  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.579584  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.575642  8654 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:02:37.580252  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.580653  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2257) took 1505 ms. Trace:
I20260709 13:02:37.168076  8469 catalog_manager.cc:5028] TS 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): tablet 9e8a9411169f407cba1b44443edc9840 (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:02:37.588111  8469 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 420173us
I20260709 13:02:37.591166  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 442) took 1319 ms. Trace:
I20260709 13:02:37.580782  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.592731  8730 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.593616  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2245) took 1869 ms. Trace:
I20260709 13:02:37.591711  8650 rpcz_store.cc:276] 0709 13:02:36.271644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.727608 (+455964us) service_pool.cc:224] Handling call
0709 13:02:37.590891 (+863283us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54001d1ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.591058 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456048896}
I20260709 13:02:37.596961  8732 rpcz_store.cc:276] 0709 13:02:35.723541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.398690 (+675149us) service_pool.cc:224] Handling call
0709 13:02:36.417829 (+ 19139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.451747 (+ 33918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.593058 (+1141311us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b54001e5fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.593379 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72054016}
I20260709 13:02:37.598273  8650 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:37.598944  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2258) took 1523 ms. Trace:
I20260709 13:02:37.599259  8650 rpcz_store.cc:276] 0709 13:02:36.075451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.597716 (+1522265us) service_pool.cc:224] Handling call
0709 13:02:37.598097 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.598681 (+   584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.598875 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.600768  8650 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:37.601336  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2259) took 1525 ms. Trace:
I20260709 13:02:37.601238  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:37.602015  8655 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:37.604023  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 781) took 1639 ms. Trace:
I20260709 13:02:37.604411  9330 rpcz_store.cc:276] 0709 13:02:35.964315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.441872 (+477557us) service_pool.cc:224] Handling call
0709 13:02:36.464148 (+ 22276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.909181 (+445033us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.603717 (+694536us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b54004de254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.603943 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":930768768}
I20260709 13:02:37.605547  8650 rpcz_store.cc:276] 0709 13:02:36.075666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.600122 (+1524456us) service_pool.cc:224] Handling call
0709 13:02:37.600552 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.601107 (+   555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.601275 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:37.606010  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:02:37.606452  8548 leader_election.cc:336] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.590s (SENT)
I20260709 13:02:37.612499  8728 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:37.614106  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:37.615164  8657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2363) took 1755 ms (client timeout 1899 ms). Trace:
W20260709 13:02:37.615576  8657 rpcz_store.cc:269] 0709 13:02:35.859580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.376893 (+517313us) service_pool.cc:224] Handling call
0709 13:02:37.614960 (+1238067us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b54000591d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.615089 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":789078784}
I20260709 13:02:37.568264  8659 rpcz_store.cc:276] 0709 13:02:36.074625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.543858 (+1469233us) service_pool.cc:224] Handling call
0709 13:02:37.566496 (+ 22638us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.567347 (+   851us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.567810 (+   463us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.617106  8657 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:37.617939 11477 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:37.618451  8659 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:02:37.648186  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2373) took 1350 ms. Trace:
I20260709 13:02:37.648624  8659 rpcz_store.cc:276] 0709 13:02:36.298217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.617483 (+1319266us) service_pool.cc:224] Handling call
0709 13:02:37.618110 (+   627us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.647824 (+ 29714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.648106 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:37.650034  8548 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.653s (SENT)
I20260709 13:02:37.653062 11661 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:37.659862  8652 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:37.660384 11686 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:37.660650  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2267) took 1266 ms. Trace:
I20260709 13:02:37.665884  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1893) took 2705 ms. Trace:
I20260709 13:02:37.666455  8726 rpcz_store.cc:276] 0709 13:02:34.960099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.709566 (+749467us) service_pool.cc:224] Handling call
0709 13:02:35.709889 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.710266 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:35.856558 (+146292us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.979410 (+122852us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:36.036552 (+ 57142us) log.cc:844] Serialized 26 byte log entry
0709 13:02:36.036695 (+   143us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:36.036725 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.036795 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:37.585660 (+1548865us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b18001c4a40. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b29522 00007f1406b2b7c1
0709 13:02:37.648336 (+ 62676us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b18001c4a40. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:37.665450 (+ 17114us) raft_consensus.cc:1685] finished
0709 13:02:37.665481 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.665670 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4327,"prepare.queue_time_us":259355,"prepare.run_wall_time_us":74,"raft.queue_time_us":19002,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":59405,"spinlock_wait_cycles":125847424,"thread_start_us":243558,"threads_started":2,"wal-append.queue_time_us":182127}
I20260709 13:02:37.669883  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.670049 11558 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.670670  8733 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:37.670778 11558 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.598202  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.674855  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.676246  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.676380  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:37.676891  8732 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:37.677706 11377 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:37.678519 11377 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.679399  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:37.620384  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2261) took 1544 ms. Trace:
I20260709 13:02:37.620775  8655 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:37.625093  8698 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.722s (SENT)
I20260709 13:02:37.682945  8657 rpcz_store.cc:276] 0709 13:02:36.076132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.616442 (+1540310us) service_pool.cc:224] Handling call
0709 13:02:37.616882 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.620016 (+  3134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.620310 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.628240  8730 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:37.675040  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 785) took 1705 ms. Trace:
I20260709 13:02:37.661059  8652 rpcz_store.cc:276] 0709 13:02:36.394239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.659171 (+1264932us) service_pool.cc:224] Handling call
0709 13:02:37.659607 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.660350 (+   743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.660577 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.632248  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 579) took 3714 ms. Trace:
I20260709 13:02:37.635862  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 787) took 1404 ms. Trace:
I20260709 13:02:37.693569  9325 rpcz_store.cc:276] 0709 13:02:36.231711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.024447 (+792736us) service_pool.cc:224] Handling call
0709 13:02:37.194410 (+169963us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.205159 (+ 10749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.635593 (+430434us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54007a99d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.635752 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":300546944}
I20260709 13:02:37.583752  8654 rpcz_store.cc:276] 0709 13:02:36.075215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.574975 (+1499760us) service_pool.cc:224] Handling call
0709 13:02:37.575422 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.580385 (+  4963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.580584 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.668197  8658 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:37.630699  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:37.631601  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2367) took 1729 ms (client timeout 1722 ms). Trace:
I20260709 13:02:37.632822  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2243) took 2965 ms. Trace:
W20260709 13:02:37.634745  8578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 571) took 3257 ms (client timeout 2300 ms). Trace:
I20260709 13:02:37.684226  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:37.690074  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 284058us
I20260709 13:02:37.700031  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:37.696409  8651 rpcz_store.cc:269] 0709 13:02:35.901880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.265715 (+363835us) service_pool.cc:224] Handling call
0709 13:02:37.631362 (+1365647us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b540026be54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.631532 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1339786624}
I20260709 13:02:37.692493  9326 rpcz_store.cc:276] 0709 13:02:33.917762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.920678 (+  2916us) service_pool.cc:224] Handling call
0709 13:02:33.921009 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.127989 (+1206980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.631851 (+2503862us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400386b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.632091 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2773370624}
W20260709 13:02:37.705742  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.697962  8578 rpcz_store.cc:269] 0709 13:02:34.376881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.377017 (+   136us) service_pool.cc:224] Handling call
0709 13:02:37.634221 (+3257204us) spinlock_profiling.cc:243] Waited 2.92 s on lock 0x7b54002c97d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.634504 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6710113536}
W20260709 13:02:37.633582  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2244) took 1910 ms (client timeout 1506 ms). Trace:
I20260709 13:02:37.710124  8731 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:37.700922  8726 raft_consensus.cc:2376] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 5. Re-sending same reply.
I20260709 13:02:37.718588  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.719228  8730 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.721998  8725 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
W20260709 13:02:37.708649  8724 rpcz_store.cc:269] 0709 13:02:35.723397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.727038 (+  3641us) service_pool.cc:224] Handling call
0709 13:02:37.633201 (+1906163us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54004727d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.633328 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1261637504}
I20260709 13:02:37.697479  8653 rpcz_store.cc:276] 0709 13:02:34.667454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.043627 (+376173us) service_pool.cc:224] Handling call
0709 13:02:35.067343 (+ 23716us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.245010 (+177667us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.632450 (+2387440us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540018b0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.632646 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":406207232}
I20260709 13:02:37.701148 11661 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:37.751008  8730 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:37.679956 11377 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.755544  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.723268 11696 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:37.758622  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 127981us
I20260709 13:02:37.758834  8730 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:37.722591 11569 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:37.722846 11608 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.741580 11638 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.757362  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:37.620659 11675 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.128s
I20260709 13:02:37.720217 11620 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 13:02:37.759932  8655 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 13:02:37.771452  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 146417us
I20260709 13:02:37.766498  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:37.769307  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.644795 11650 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:37.775898  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.778815  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.779469  8730 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.777583  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:37.644134 11486 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.123s
I20260709 13:02:37.787700  8698 leader_election.cc:400] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:02:37.784199  8724 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.788208  8698 leader_election.cc:403] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 13:02:37.789964  8732 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:37.765281 11594 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.793449  8726 raft_consensus.cc:2410] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:37.794253  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:37.791989  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.692080  9322 rpcz_store.cc:276] 0709 13:02:35.969245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.971925 (+1002680us) service_pool.cc:224] Handling call
0709 13:02:36.986598 (+ 14673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.006303 (+ 19705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.674813 (+668510us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b5400519350. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.674984 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30483840}
I20260709 13:02:37.796401  8732 raft_consensus.cc:2410] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:37.800524  8584 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:37.798934  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:37.801586  8697 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.716s (SENT)
I20260709 13:02:37.808660  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.810010  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.771162 11608 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.780501 11638 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.799762 11486 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:37.829816  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2249) took 1788 ms. Trace:
I20260709 13:02:37.834288 11486 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.834681  8727 rpcz_store.cc:276] 0709 13:02:36.041174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.739430 (+698256us) service_pool.cc:224] Handling call
0709 13:02:36.739973 (+   543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.936138 (+196165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.829536 (+893398us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54001e73d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.829745 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":449655680}
I20260709 13:02:37.835465 11486 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.836496  8727 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 13:02:37.821834  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68652.9us
I20260709 13:02:37.830320  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.822257  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.829799  8581 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:37.826973  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.847559  8576 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:37.806945  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.795428  8650 raft_consensus.cc:2955] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:02:37.813210  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.828589  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:37.824201 11602 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:37.842912  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2235) took 2120 ms. Trace:
I20260709 13:02:37.797354 11632 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:37.851606  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:37.805359 11643 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:37.852417  8729 rpcz_store.cc:276] 0709 13:02:35.721992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.286886 (+564894us) service_pool.cc:224] Handling call
0709 13:02:36.287222 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:36.587979 (+300757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.836136 (+1248157us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54000eb354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:37.842720 (+  6584us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551395200}
I20260709 13:02:37.851887  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.821707 11587 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.818022 11696 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:37.896354  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.819299 11650 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:37.862556  8725 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:37.860522  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:37.814810 11593 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:37.909754  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 393838us
I20260709 13:02:37.863581  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:37.814188 11623 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.826920 11558 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.818985 11679 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.901067  8728 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:37.816032 11604 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:02:37.904241 11579 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:37.904415  8582 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:37.906142  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.906730  8698 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:37.907024  8727 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:37.907088  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1908) took 1058 ms. Trace:
I20260709 13:02:37.923620  8729 rpcz_store.cc:276] 0709 13:02:36.848692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.861267 (+1012575us) service_pool.cc:224] Handling call
0709 13:02:37.906507 (+ 45240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.906731 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:37.906757 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:37.906807 (+    50us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:37.906846 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.906904 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.907009 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:37.908427 11632 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.906221  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:37.909937  9329 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:37.858294  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:37.933081  8653 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:37.914179  9325 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
W20260709 13:02:37.916641  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.917683  8728 pending_rounds.cc:85] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:37.852284 11602 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:37.912997  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:37.919278 11679 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:37.931437  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:37.914286 11594 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.935151  8728 pending_rounds.cc:107] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:37.910181 11593 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:37.955639  8728 raft_consensus.cc:2889] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:37.948932  8655 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:02:37.949179  8582 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:37.950516  8584 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:37.951359  8732 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:37.921190 11623 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:37.913885 11643 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:37.952229 11608 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:37.952826  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 783) took 1908 ms (client timeout 1590 ms). Trace:
W20260709 13:02:37.953785  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47762.6us
I20260709 13:02:37.931605 11486 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.985442  8724 raft_consensus.cc:2410] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:02:37.969990  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1905) took 1122 ms. Trace:
I20260709 13:02:37.986044  8728 rpcz_store.cc:276] 0709 13:02:36.847554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.974475 (+126921us) service_pool.cc:224] Handling call
0709 13:02:36.975136 (+   661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.612284 (+637148us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5800778dd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b720aa 00007f1409bf4447 00007f1409c0cfe7 00007f1409c0e7de 00007f1409c18b49 00007f1409c19416 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4
0709 13:02:37.969535 (+357251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.969908 (+   373us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":443935104}
I20260709 13:02:37.956637  8655 pending_rounds.cc:85] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 3
I20260709 13:02:37.987326  8655 pending_rounds.cc:107] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:02:37.987690  8655 raft_consensus.cc:2889] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:37.969534  8730 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:37.990504  8733 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:37.955682 11675 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:37.997259  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:38.001346  8580 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 13:02:37.997939  8733 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:38.002720 11638 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:37.919377 11587 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.006994  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:38.010500  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.010632  8728 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:38.011170 11607 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 13:02:37.972752 11558 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:38.012339  8730 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:02:38.014421  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 175246us
I20260709 13:02:37.990485 11579 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:37.958595 11594 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.016786 11602 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.023206 11587 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:38.022920 11579 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:37.985345  9324 rpcz_store.cc:269] 0709 13:02:36.044082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.271834 (+227752us) service_pool.cc:224] Handling call
0709 13:02:37.952560 (+1680726us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54002779d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:37.952742 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1056091520}
I20260709 13:02:38.023528 11593 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.025144 11638 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.025864 11643 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.030004  8728 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:38.042802  8728 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:38.030431  8582 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:38.043493  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:38.030700  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 75992.5us
I20260709 13:02:38.045634  8729 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:38.037933  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:38.054277  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.062386  8730 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
W20260709 13:02:38.063505  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.063654  8730 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260709 13:02:38.064033  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.061838 11593 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:38.075239  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54901.4us
W20260709 13:02:38.089692  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 328247us
I20260709 13:02:38.093231  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2246) took 2021 ms. Trace:
I20260709 13:02:38.093859  8658 rpcz_store.cc:276] 0709 13:02:36.072215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.413974 (+1341759us) service_pool.cc:224] Handling call
0709 13:02:38.092995 (+679021us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400470754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:38.093151 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":555099392}
I20260709 13:02:38.097822  8582 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:38.074138 11602 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:38.102828  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.056705 11638 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.104359  8652 pending_rounds.cc:85] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 2
I20260709 13:02:38.106906  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.118157  8652 pending_rounds.cc:107] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 13:02:38.118496  8652 raft_consensus.cc:2889] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:38.115747 11732 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 13:02:38.114516 11693 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 13:02:38.114055 11590 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:38.121793  8727 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260709 13:02:38.091706 11696 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:38.122766  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.127143  8584 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:38.123742  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:38.130434  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.134408  9296 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:38.134527  8658 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:38.134960  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.135720  8654 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:38.136786 11608 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:38.137472 11590 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:38.138497 11590 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.141942  8727 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:38.143416  8727 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 13:02:38.144568 11735 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:02:38.145836 11696 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:38.190289  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56190.2us
I20260709 13:02:38.192126  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:38.204346  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 138898us
W20260709 13:02:38.219817  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.228003  8653 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 13:02:38.229668  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 78166.2us
I20260709 13:02:38.146317 11579 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:38.166045 11608 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:38.235967  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.247562  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:38.255513  8659 raft_consensus.cc:1217] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Deduplicated request from leader. Original: 2.5->[3.6-3.6]   Dedup: 3.6->[]
I20260709 13:02:38.256155  8659 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:38.257855 11558 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.233309 11731 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:38.261420 11733 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 13:02:38.265201  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2254) took 1159 ms. Trace:
I20260709 13:02:38.263422  9326 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 13:02:38.259666 11736 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.220s	user 0.001s	sys 0.001s
I20260709 13:02:38.260910 11558 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:38.294893  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48984.1us
W20260709 13:02:38.301815  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 196011us
W20260709 13:02:38.304272  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58665.6us
I20260709 13:02:38.304297  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:38.265666 11679 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:38.309058 11594 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:38.327087  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:38.332746  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:38.294690 11604 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:38.261693 11579 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
W20260709 13:02:38.344151  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 96638.3us
W20260709 13:02:38.367959  8656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 437) took 2132 ms (client timeout 2194 ms). Trace:
I20260709 13:02:38.332685 11569 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:38.332226 11739 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.334695 11693 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:38.371222  8656 rpcz_store.cc:269] 0709 13:02:36.235995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.068693 (+832698us) service_pool.cc:224] Handling call
0709 13:02:38.367745 (+1299052us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:38.367896 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252096768}
I20260709 13:02:38.309908 11686 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260709 13:02:38.400573 11558 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:38.401928 11753 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:02:38.371735 11705 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:38.390710 11693 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.315132 11620 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.071s
I20260709 13:02:38.409314  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2260) took 2328 ms. Trace:
I20260709 13:02:38.409750  8650 rpcz_store.cc:276] 0709 13:02:36.075895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.606434 (+1530539us) service_pool.cc:224] Handling call
0709 13:02:37.607089 (+   655us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:37.607731 (+   642us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:37.607760 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:37.608886 (+  1126us) log.cc:844] Serialized 513 byte log entry
0709 13:02:37.768752 (+159866us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:38.197001 (+428249us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.232498 (+ 35497us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.232608 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:38.404109 (+171501us) raft_consensus.cc:1685] finished
0709 13:02:38.404129 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.404265 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49831,"raft.queue_time_us":37074,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":9178,"thread_start_us":159466,"threads_started":1,"wal-append.queue_time_us":391598}
I20260709 13:02:38.383827 11578 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:38.428485  8654 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:38.433501  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:38.459251  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 126568us
W20260709 13:02:38.461676  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 193647us
I20260709 13:02:38.419093 11638 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:38.458742 11654 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:38.390506 11739 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.386399 11608 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.483713 11623 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:38.493928 11742 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
W20260709 13:02:38.494526 11752 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
W20260709 13:02:38.498451 11750 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 13:02:38.499747 11616 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:38.500406 11747 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 13:02:38.475859 11679 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:38.526412  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.529541  8654 raft_consensus.cc:2410] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:02:38.527316  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 113047us
W20260709 13:02:38.521698 11751 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
W20260709 13:02:38.532611  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 230012us
W20260709 13:02:38.542605  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.542441  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.543779  8650 raft_consensus.cc:2410] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:02:38.550513  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.551220  8650 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:38.270375  8733 rpcz_store.cc:276] 0709 13:02:37.099586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.002520 (+902934us) service_pool.cc:224] Handling call
0709 13:02:38.002822 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.003047 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:38.003342 (+   295us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.258401 (+255059us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.258587 (+   186us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:38.258646 (+    59us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:38.258728 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.258801 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.258941 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107885,"prepare.queue_time_us":19886,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":1913,"raft.queue_time_us":94826,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":11326,"spinlock_wait_cycles":44579456,"thread_start_us":153113,"threads_started":1}
I20260709 13:02:38.552114  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.552616  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:38.552917  8650 raft_consensus.cc:2410] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:38.553740  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.554034  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.554579  8656 raft_consensus.cc:2410] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:38.555377  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.556196  8656 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:38.556996  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.557729  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.558159 11693 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:38.558324  8658 raft_consensus.cc:2376] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6. Re-sending same reply.
I20260709 13:02:38.555315  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.559711  8650 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:38.559878  8656 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:38.561687  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:38.562096 11755 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.091s	user 0.003s	sys 0.000s
I20260709 13:02:38.562328  8653 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:38.563143  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.561038  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.565549  8658 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:38.566722  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1914) took 1516 ms. Trace:
I20260709 13:02:38.567727  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:38.567173  8732 rpcz_store.cc:276] 0709 13:02:37.049948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.974058 (+924110us) service_pool.cc:224] Handling call
0709 13:02:38.006473 (+ 32415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.013516 (+  7043us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:38.037668 (+ 24152us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.566335 (+528667us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:38.566395 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:38.566457 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.566540 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.566657 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55483,"prepare.queue_time_us":17594,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":67455,"spinlock_wait_cycles":6069120,"thread_start_us":460927,"threads_started":2,"wal-append.queue_time_us":216143}
I20260709 13:02:38.572489  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2264) took 1350 ms. Trace:
W20260709 13:02:38.575078  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.573594  8730 rpcz_store.cc:276] 0709 13:02:37.222323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.068351 (+846028us) service_pool.cc:224] Handling call
0709 13:02:38.068735 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:38.069008 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:38.069047 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:38.090424 (+ 21377us) log.cc:844] Serialized 26 byte log entry
0709 13:02:38.293969 (+203545us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:38.294044 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.294166 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:38.561455 (+267289us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:38.562978 (+  1523us) raft_consensus.cc:1685] finished
0709 13:02:38.563000 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.572422 (+  9422us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37518080,"thread_start_us":144208,"threads_started":1,"wal-append.queue_time_us":290709}
I20260709 13:02:38.535686 11705 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:38.579305  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:38.580179  8733 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:38.583519  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.587848  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.588537  8654 raft_consensus.cc:2376] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5. Re-sending same reply.
I20260709 13:02:38.590915  8732 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:38.537214 11616 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.594046  8733 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:38.570569 11578 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:38.572417 11687 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.599370  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.595445 11766 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:38.602993 11675 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.591s	user 0.003s	sys 0.008s
I20260709 13:02:38.606055 11731 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:38.595760 11679 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:02:38.608088  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.608090  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47922.7us
I20260709 13:02:38.608824  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1906) took 1760 ms. Trace:
I20260709 13:02:38.624171  8725 rpcz_store.cc:276] 0709 13:02:36.848002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.815762 (+967760us) service_pool.cc:224] Handling call
0709 13:02:37.816179 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.902324 (+ 86145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.608579 (+706255us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b5400229f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:38.608768 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106000640}
W20260709 13:02:38.609720  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:38.611511  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1912) took 1682 ms. Trace:
I20260709 13:02:38.628288  8729 rpcz_store.cc:276] 0709 13:02:36.929565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.950505 (+1020940us) service_pool.cc:224] Handling call
0709 13:02:37.950857 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.046310 (+ 95453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.611258 (+564948us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b54000898d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:38.611441 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":212118272}
I20260709 13:02:38.613020 11738 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:38.616899  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:38.626590  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.638093 11705 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.639258  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2270) took 2244 ms. Trace:
I20260709 13:02:38.640776 11739 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:38.639674  8657 rpcz_store.cc:276] 0709 13:02:36.395109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.701855 (+1306746us) service_pool.cc:224] Handling call
0709 13:02:37.753497 (+ 51642us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.783940 (+ 30443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:37.783974 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:37.803291 (+ 19317us) log.cc:844] Serialized 26 byte log entry
0709 13:02:38.101277 (+297986us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:38.329750 (+228473us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.330863 (+  1113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.330950 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:38.639013 (+308063us) raft_consensus.cc:1685] finished
0709 13:02:38.639033 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.639193 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7326,"prepare.queue_time_us":35638,"prepare.run_cpu_time_us":20,"prepare.run_wall_time_us":21,"raft.queue_time_us":14599,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":34810,"thread_start_us":478444,"threads_started":2,"wal-append.queue_time_us":752245}
I20260709 13:02:38.641675 11768 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:38.642120 11768 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.642458 11768 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:38.625811 11687 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.647584 11769 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:38.648933 11739 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:38.649443 11739 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:38.622630 11766 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.650593 11616 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:38.650657 11739 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.625568 11578 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.651978 11766 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:38.652269 11739 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:38.653020  8656 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 4: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:38.626106 11675 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:38.658206  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:38.660624  8658 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
W20260709 13:02:38.667937  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.669275 11593 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.003s
I20260709 13:02:38.631630 11738 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:38.647584 11687 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:38.686021  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 60406.9us
I20260709 13:02:38.674831 11675 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:02:38.691061  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54356us
I20260709 13:02:38.693979 11590 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:38.694120 11602 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:38.694612 11590 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.695034 11590 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:38.695200 11602 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:38.695684 11602 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:38.696801 11704 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:38.704937  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2261) took 1557 ms. Trace:
I20260709 13:02:38.706983  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2256) took 1606 ms. Trace:
I20260709 13:02:38.707376  8724 rpcz_store.cc:276] 0709 13:02:37.100261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.009618 (+909357us) service_pool.cc:224] Handling call
0709 13:02:38.010051 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.030538 (+ 20487us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:38.198589 (+168051us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.706649 (+508060us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:38.706707 (+    58us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:38.706752 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.706816 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.706930 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":370600,"prepare.queue_time_us":26872,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":7528,"spinlock_wait_cycles":28504960,"thread_start_us":258353,"threads_started":2,"wal-append.queue_time_us":283352}
W20260709 13:02:38.709141  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 363595us
W20260709 13:02:38.711381  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.659863 11693 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 13:02:38.705365  8728 rpcz_store.cc:276] 0709 13:02:37.147528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.043549 (+896021us) service_pool.cc:224] Handling call
0709 13:02:38.059529 (+ 15980us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.063158 (+  3629us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:38.090298 (+ 27140us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.704405 (+614107us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.704578 (+   173us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:38.704620 (+    42us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:38.704670 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.704737 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.704857 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":368547,"prepare.queue_time_us":33467,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":37,"thread_start_us":217249,"threads_started":1,"wal-append.queue_time_us":488640}
I20260709 13:02:38.713009  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.715984  8584 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:38.717440  8653 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:38.719179  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:38.719893  9325 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:38.720325  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.721072  8658 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:38.727813  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:38.728739  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2266) took 1112 ms. Trace:
I20260709 13:02:38.729765  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.730940  8655 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:38.732463  8584 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:38.729266  8727 rpcz_store.cc:276] 0709 13:02:37.616747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.145333 (+528586us) service_pool.cc:224] Handling call
0709 13:02:38.153068 (+  7735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.197693 (+ 44625us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:38.257629 (+ 59936us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.728218 (+470589us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.728376 (+   158us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:38.728415 (+    39us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:38.728463 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.728522 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.728660 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":254176,"prepare.queue_time_us":25668,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":54,"thread_start_us":164584,"threads_started":1,"wal-append.queue_time_us":269390}
I20260709 13:02:38.746151  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.747375  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:38.749536  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43768.9us
I20260709 13:02:38.755075  8654 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: [2.5-5.6]
I20260709 13:02:38.713618 11704 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:38.748621  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:38.767572  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:38.770886  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:38.761601 11486 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.765s	user 0.013s	sys 0.001s
I20260709 13:02:38.775175  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:38.696938 11602 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.798163  8581 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:02:38.798229  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98975.2us
I20260709 13:02:38.802664  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2270) took 1183 ms. Trace:
I20260709 13:02:38.801440  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:38.808269  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 193228us
I20260709 13:02:38.812907  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:38.809247  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2380) took 1886 ms. Trace:
I20260709 13:02:38.810618  8730 rpcz_store.cc:276] 0709 13:02:37.619116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.577474 (+958358us) service_pool.cc:224] Handling call
0709 13:02:38.585459 (+  7985us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.624062 (+ 38603us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:38.634529 (+ 10467us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.802261 (+167732us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:38.802326 (+    65us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:38.802380 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.802451 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.802598 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54301,"prepare.queue_time_us":10350,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"thread_start_us":100258,"threads_started":1,"wal-append.queue_time_us":142518}
I20260709 13:02:38.814284  8659 rpcz_store.cc:276] 0709 13:02:36.922835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.690551 (+767716us) service_pool.cc:224] Handling call
0709 13:02:37.720654 (+ 30103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:38.256533 (+535879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.808948 (+552415us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54004704d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:38.809169 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1199688960}
I20260709 13:02:38.772432 11731 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
W20260709 13:02:38.827441  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.831673  8658 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:38.826601  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.832839  8650 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:38.830005  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:38.826934  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:38.769649 11558 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
W20260709 13:02:38.824222  8696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.853s (SENT) [suppressed 56 similar messages]
I20260709 13:02:38.811172 11696 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.841581  9322 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 13:02:38.840126  8657 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:38.839985  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:38.842521  9325 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:38.846010  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.842000  9322 pending_rounds.cc:85] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 7
I20260709 13:02:38.849071  9322 pending_rounds.cc:107] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 13:02:38.849846  9322 raft_consensus.cc:2889] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:38.851352  8730 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:38.851814  8730 pending_rounds.cc:85] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:38.852036  8730 pending_rounds.cc:107] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:38.852254  8730 raft_consensus.cc:2889] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:38.829120 11693 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.834048 11632 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:38.866798  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:38.867791  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:38.868788  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 104368us
I20260709 13:02:38.777267 11738 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.895010  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:38.895413  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:38.895810  8724 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:38.896073  8731 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:38.825627 11771 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 13:02:38.836139 11604 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:38.929522  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:38.932438  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64730us
I20260709 13:02:38.933713  9324 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:38.935319 11744 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.444s	user 0.001s	sys 0.000s
W20260709 13:02:38.936089  8548 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.760s (SENT)
I20260709 13:02:38.936614  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:38.900501  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:38.938508  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1916) took 1864 ms. Trace:
I20260709 13:02:38.938997  8726 rpcz_store.cc:276] 0709 13:02:37.073751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.804675 (+730924us) service_pool.cc:224] Handling call
0709 13:02:38.938286 (+1133611us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54001e5fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:38.938440 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2199618432}
I20260709 13:02:38.939643  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:38.938094  8729 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 2.
I20260709 13:02:38.940590  8730 raft_consensus.cc:2376] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4. Re-sending same reply.
I20260709 13:02:38.941684  8577 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:38.942035  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 99219.3us
I20260709 13:02:38.944846  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 594) took 1184 ms. Trace:
I20260709 13:02:38.846622 11623 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 13:02:38.897003 11602 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:38.889768 11679 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:38.951242  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:38.947678  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:38.954154  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:38.955170  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:38.956727  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.945303  8575 rpcz_store.cc:276] 0709 13:02:37.761027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.766379 (+  5352us) service_pool.cc:224] Handling call
0709 13:02:37.785203 (+ 18824us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:37.798396 (+ 13193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:37.798440 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:37.809859 (+ 11419us) log.cc:844] Serialized 26 byte log entry
0709 13:02:38.058148 (+248289us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:38.058186 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.058263 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:38.934652 (+876389us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:38.940039 (+  5387us) raft_consensus.cc:1685] finished
0709 13:02:38.940061 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.944782 (+  4721us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10672,"raft.queue_time_us":10861,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":29343,"spinlock_wait_cycles":8862592,"thread_start_us":161538,"threads_started":1,"wal-append.queue_time_us":557955}
W20260709 13:02:38.960640  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.890564 11632 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:38.881493 11693 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.901973 11486 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.826166 11704 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.940722 11607 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:38.858796 11696 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:38.945062 11761 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.463s	user 0.003s	sys 0.000s
I20260709 13:02:38.958046  8732 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:38.978888  9325 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:38.980702  8732 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:38.929829 11778 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.972165  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:38.974589  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 93563.7us
I20260709 13:02:38.975243  8650 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:38.974265 11604 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:38.983669 11787 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:38.985553  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:38.985266 11778 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:38.986258  9324 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:38.988813 11778 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:38.990262  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.992754  9322 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:38.976217 11486 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:38.995568 11773 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
W20260709 13:02:38.997282 11775 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 13:02:38.990208  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:38.974944 11738 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:39.002256  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62441.2us
I20260709 13:02:38.997057 11486 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:38.977762 11785 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:38.982617 11607 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.982038 11558 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:39.005903  8732 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.018682  9322 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:38.999017 11704 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:39.089608  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82493us
W20260709 13:02:39.096263  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.092456  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.112322 11788 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:39.015090 11607 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:39.113301 11594 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 13:02:39.127015 11745 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:39.130755 11693 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.015863 11779 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:39.131108 11745 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:39.131449 11779 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:39.131596 11745 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:39.131916 11779 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:39.013451 11785 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:39.023409 11558 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:39.092927 11692 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.093287 11623 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:39.132670 11745 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:39.167182  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 191551us
W20260709 13:02:39.168763  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.169147  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.117511 11783 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:39.171669  9322 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 13:02:39.149706 11558 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:39.174278  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 84700.9us
W20260709 13:02:39.162909 11782 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 13:02:39.179957  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:39.180851  9322 raft_consensus.cc:2376] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 6. Re-sending same reply.
I20260709 13:02:39.157194 11692 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.188162 11783 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:02:39.187716  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.197863  9322 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:02:39.146553 11632 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:39.199026  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 217051us
I20260709 13:02:39.203096 11617 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.203600 11617 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.206156 11617 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:39.192303 11793 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 13:02:39.189685 11745 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:39.221825  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.222673  9324 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:39.199059 11737 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.112s	user 0.002s	sys 0.005s
I20260709 13:02:39.213446 11785 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.200978 11788 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:39.246167  8727 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:02:39.240216  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:39.253715  8732 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:39.288995  8731 raft_consensus.cc:2376] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 7. Re-sending same reply.
I20260709 13:02:39.251334  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1787) took 1351 ms. Trace:
I20260709 13:02:38.637635  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:02:39.230165 11784 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
W20260709 13:02:39.259410 11768 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.616s	user 0.006s	sys 0.005s
I20260709 13:02:39.300437  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:39.301817  8654 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:39.248202 11692 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:39.296530  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:39.230470 11604 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.257217 11696 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.241617 11795 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.235489 11607 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.232611 11558 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.287643 11788 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:39.313024  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.299247 11578 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.347803  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:39.291532  8576 rpcz_store.cc:276] 0709 13:02:37.900033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.900208 (+   175us) service_pool.cc:224] Handling call
0709 13:02:37.907925 (+  7717us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:38.008923 (+100998us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:38.008951 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:38.019061 (+ 10110us) log.cc:844] Serialized 26 byte log entry
0709 13:02:38.128460 (+109399us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:38.189353 (+ 60893us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.201980 (+ 12627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.250962 (+1048982us) spinlock_profiling.cc:243] Waited 93.6 ms on lock 0x7b5400440cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:39.251009 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:39.251051 (+    42us) raft_consensus.cc:1685] finished
0709 13:02:39.251140 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.251277 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5341,"prepare.queue_time_us":9091,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":42362,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":14189,"spinlock_wait_cycles":215338752,"thread_start_us":155121,"threads_started":2,"wal-append.queue_time_us":305262}
I20260709 13:02:39.292043 11617 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.332803  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.357179  9322 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:39.303990 11768 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.348483  8731 raft_consensus.cc:2376] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4. Re-sending same reply.
W20260709 13:02:39.353997  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.345875  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 388036us
I20260709 13:02:39.331367 11660 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.295836 11779 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.368470  9322 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:39.378019  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 441891us
I20260709 13:02:39.341737 11731 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:39.345072 11785 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:39.369048  8696 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.611s (SENT)
W20260709 13:02:39.370945  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.391172  8657 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:39.395102  8732 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:39.331123 11604 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.397969  9322 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:02:39.400521  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.403415  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 226367us
I20260709 13:02:39.405800  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.395761  8732 pending_rounds.cc:85] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:39.410548  8732 pending_rounds.cc:107] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:39.410902  8732 raft_consensus.cc:2889] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:39.412849  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.424450  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.406879  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:39.363039 11617 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:39.443061 11790 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:02:39.444571 11617 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.445251 11798 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:39.441480 11696 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.446601 11696 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.447935 11768 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.346716 11795 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.359731 11578 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.439705  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:39.449379  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44518.2us
I20260709 13:02:39.444317 11788 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.483412  9322 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:39.395675 11731 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.485824  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1784) took 1666 ms. Trace:
I20260709 13:02:39.448895 11608 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:39.496115  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 101513us
I20260709 13:02:39.379086 11660 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.499840 11791 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:39.501346 11696 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:39.486773 11799 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:39.502466 11799 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.502876 11799 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:39.500272 11660 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:39.486979  8577 rpcz_store.cc:276] 0709 13:02:37.819620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.859262 (+ 39642us) service_pool.cc:224] Handling call
0709 13:02:38.941512 (+1082250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.944532 (+  3020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.485600 (+541068us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000b6650. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:39.485762 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2485673216}
W20260709 13:02:39.506457  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 107397us
W20260709 13:02:39.494704 11590 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.799s	user 0.010s	sys 0.005s
I20260709 13:02:39.515381  9322 raft_consensus.cc:1217] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[3.6-3.6]   Dedup: 3.6->[]
I20260709 13:02:39.515991  9322 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:39.524497  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:39.525609 11798 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:39.526049 11798 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:39.524519 11607 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:39.527137 11798 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.528609 11679 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:39.529220 11679 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:39.532055  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.534488  8659 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:39.512197 11788 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:39.536868  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.537613  9325 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:39.542667 11787 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:39.544395 11787 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:39.545718  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.546576 11791 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:39.547624 11794 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:39.548638 11602 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:39.552544 11602 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:39.553956 11602 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.555881 11602 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:39.558425  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.559360 11794 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:39.549425 11787 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:02:39.560585  8548 leader_election.cc:336] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.615s (SENT)
I20260709 13:02:39.561213  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:39.569756 11793 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:39.571938 11794 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:39.573179 11787 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:39.573911 11787 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:39.577092 11787 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.579248 11787 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:39.580911 11768 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.575352  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:39.588184 11607 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:39.588428 11794 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:39.589038 11794 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.588748 11607 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.589576 11794 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:39.591291 11607 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.595299 11793 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:39.600525 11632 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:39.602355  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2277) took 1473 ms. Trace:
I20260709 13:02:39.607191  8651 rpcz_store.cc:276] 0709 13:02:38.129221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.837137 (+707916us) service_pool.cc:224] Handling call
0709 13:02:39.322983 (+485846us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.323386 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:39.323420 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:39.333185 (+  9765us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:39.333255 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.333363 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.602152 (+268789us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b540018e7d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:39.602288 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":14223,"spinlock_wait_cycles":1068818176}
I20260709 13:02:39.612824 11578 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:39.615480 11558 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.606554  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:39.584754  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:39.585196  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.625938  9322 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 4: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:02:39.626689  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.629344  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:39.629832  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:39.634645  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48399.4us
I20260709 13:02:39.636165  8625 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:39.584764  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.630064  9325 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:39.638841  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.629801  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.640864  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.508368 11608 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.641913  9328 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:39.644246  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:39.645025  8650 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:39.652885  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.656948  9323 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:39.658133  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:39.658257  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.659021  9323 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:39.668290  9325 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 13:02:39.669056  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62972.2us
I20260709 13:02:39.586678 11604 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.520489 11590 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.675361  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.687378  9323 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:39.655632 11783 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
W20260709 13:02:39.691283 11801 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 13:02:39.677758 11675 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:39.695245  8650 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
W20260709 13:02:39.695587  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.695447  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.698362  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:39.701820  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 183610us
I20260709 13:02:39.704670  9322 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:39.704542  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.656945 11569 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:39.707940  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.622485 11768 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.675026  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.651661 11791 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 13:02:39.698361 11692 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.676265 11738 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:02:39.723701  9325 raft_consensus.cc:2410] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:39.724545  9327 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:39.729208  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 69274.3us
I20260709 13:02:39.676034 11578 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 13:02:39.724859  9330 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:39.696167 11593 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:02:39.677392 11617 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
W20260709 13:02:39.738281  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.751034  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.754055  9297 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:39.755210  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:39.756178  9327 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:39.744899  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:39.742511 11632 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:39.781378  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.778609  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:39.787717  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 138895us
I20260709 13:02:39.735222 11739 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20260709 13:02:39.746632 11704 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:02:39.798869  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.783015 11579 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:39.746264 11692 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.799654 11739 consensus_queue.cc:1243] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:39.793378 11558 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 13:02:39.796221 11604 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:39.806608 11786 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:39.810525 11808 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:39.811146 11807 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:39.812326 11731 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:39.817859  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.783989 11779 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:39.783347 11632 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:39.836318  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.820349 11623 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.851956 11795 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.835034 11786 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:39.840555 11785 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.841048 11578 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:39.797864 11811 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
W20260709 13:02:39.864984  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47220.5us
I20260709 13:02:39.866209  8697 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:39.852008 11632 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:39.854960 11807 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:39.858860 11608 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:39.870200 11578 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.867949 11795 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:39.869797 11579 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:39.876725 11807 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:39.883092  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 138303us
I20260709 13:02:39.899163  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:39.900885  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:39.878508 11704 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:39.862076 11623 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:39.905949  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 115200us
I20260709 13:02:39.860323 11817 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 13:02:39.910209  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:39.907509  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 92141.5us
W20260709 13:02:39.905890  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 145730us
I20260709 13:02:39.910897  8728 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:39.911897  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:39.912788 11808 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:39.913308  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:39.913769  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:39.914306 11813 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:39.914467  8732 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:39.915143 11813 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:39.916355 11817 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:39.916124 11813 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:39.917449  8731 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:39.918062  8727 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:39.917506 11817 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:39.880950 11806 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:39.920518  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:39.871246 11785 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:39.918885  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:39.928028  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.933763  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:39.932049  8724 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:39.934947  8730 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:39.936074  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:39.936228  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:39.936978  8725 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:39.903842 11783 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:39.947417  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:39.940665  8733 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 13:02:39.926795 11617 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:39.905218 11739 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:39.951205  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:39.907127 11704 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.952253  8724 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:39.968456  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:39.969383  8730 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:39.951118  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:39.971293  8729 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:39.972461  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53653.9us
I20260709 13:02:39.927999 11768 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:39.973651  8657 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:39.953723 11617 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.928795 11808 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:39.979781  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:39.981170  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 61597.3us
W20260709 13:02:39.981645  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 252729us
I20260709 13:02:39.929339 11817 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:39.976356 11593 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:40.005365  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 206560us
I20260709 13:02:40.003242  8725 raft_consensus.cc:2410] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:39.953884 11739 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.953612 11660 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:40.033223  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.974229 11704 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:40.040021  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 60317.3us
I20260709 13:02:39.978716 11617 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:39.977723 11675 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:39.967957 11783 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:40.057986  8729 pending_rounds.cc:85] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 2
I20260709 13:02:40.087716  8729 pending_rounds.cc:107] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 13:02:40.088107  8729 raft_consensus.cc:2889] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:40.058555  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.088527  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.058791 11794 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.059216 11623 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.034418 11739 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:40.030177 11808 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.061009  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.063599  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.062280  8725 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:39.902015 11786 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.103925  8582 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:40.098516  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.072242 11799 consensus_meta.cc:287] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.569s	user 0.013s	sys 0.000s
I20260709 13:02:40.042023 11768 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.037606 11593 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.097092 11820 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:40.077862  8580 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:40.078689 11578 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.081751 11692 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.081959  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:40.095883 11821 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:40.140573  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.107185  8576 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:40.108897 11786 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:39.945665 11806 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.092309 11817 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.110925 11675 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.091303 11824 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:02:40.119406 11604 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:40.115849 11783 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.121786 11799 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.122696 11704 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.090245 11816 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.071521  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.135807  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.136076  8581 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:40.136312  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:40.135545 11593 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.096448 11826 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 13:02:40.138077  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.139348  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.170250  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.177631  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:40.178702  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 96799.2us
I20260709 13:02:40.179759  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:40.150096  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 115249us
W20260709 13:02:40.125329 11788 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.539s	user 0.008s	sys 0.004s
I20260709 13:02:40.127202 11794 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.129070 11808 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.193791  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:40.194622  9330 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:40.194677  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:40.195376 11704 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.195781  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:40.195878  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:40.196413  9325 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.196730  9328 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:40.197851  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:40.193831  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.130895 11569 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:40.199589 11569 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:40.200536 11632 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:40.135838  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.129949 11785 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:40.136761 11820 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.185546  8578 raft_consensus.cc:2368] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:02:40.186358 11731 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:40.186255  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.190831 11593 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.191422  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:40.192788 11806 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.156623  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.200639 11569 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.159433  8725 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:40.230073  8580 raft_consensus.cc:2368] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:02:40.201689  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:40.202143  8583 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:40.203904  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:40.133854 11768 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.231540 11692 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:40.157338 11804 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:40.158567 11811 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.208683  8579 raft_consensus.cc:2410] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:40.195402  9322 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:40.217478  9325 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:40.206207  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:40.201743 11808 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:40.227727  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:40.231779  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.255903  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.206750  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.250984  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:40.256847  8732 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:40.176858 11617 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.228999 11822 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:40.257315  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:40.254411  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48288.6us
I20260709 13:02:40.223475 11731 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:40.261811  8581 raft_consensus.cc:2368] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:02:40.229352 11817 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:40.181474 11816 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.263669  8725 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:40.223717 11820 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:40.269857  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66021.8us
I20260709 13:02:40.271111  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:40.267819  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.195952 11704 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:40.291229  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.258982 11791 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:40.226442 11593 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.259096 11824 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:40.294627  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:40.266525 11822 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.296217  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.303653  8583 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:40.303894  8732 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:40.260700 11804 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:40.267254 11806 raft_consensus.cc:2764] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 13:02:40.305233  8732 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:02:40.269909 11569 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:40.225204 11825 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260709 13:02:40.309185  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:40.310822  8732 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260709 13:02:40.311758  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.318092  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:40.320813  8582 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:40.258216 11811 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.324051  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.323278  8732 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:40.297596 11808 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:40.294646 11791 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:40.307101 11822 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:40.338557  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:40.369112  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 151818us
I20260709 13:02:40.369180 11660 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:40.374688 11821 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:40.375056 11791 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.324434 11798 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.385270 11738 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:40.385514  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.385859 11738 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:40.386333  8576 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:40.391598  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:40.388353  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:40.394351 11809 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.737s	user 0.006s	sys 0.001s
I20260709 13:02:40.395938 11816 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.273249 11704 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.396874 11799 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.395869 11826 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:40.392970  8726 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:40.398305  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:40.422272  8724 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.314960 11807 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:40.393559 11768 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.403331  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:40.404107  8580 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:40.400267  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.338212 11837 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:40.409768  8732 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:40.417145  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 111640us
I20260709 13:02:40.442324  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.339864 11806 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:40.410663 11805 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.008s	sys 0.019s
W20260709 13:02:40.423234  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.434504  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:40.386929 11738 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.439142  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.386214 11632 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:40.446869  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 188415us
W20260709 13:02:40.457958  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54677.4us
I20260709 13:02:40.433327 11816 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.400846 11739 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.512993  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:40.436698 11813 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:40.532358  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.540102  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.541188  8582 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:40.514479  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.542364  8580 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:40.542461  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.543320  8582 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:40.544391  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.527886  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.548220  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 606) took 1147 ms. Trace:
I20260709 13:02:40.549994  8584 rpcz_store.cc:276] 0709 13:02:39.401255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.401368 (+   113us) service_pool.cc:224] Handling call
0709 13:02:39.414949 (+ 13581us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:39.486839 (+ 71890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:39.486879 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:39.498181 (+ 11302us) log.cc:844] Serialized 26 byte log entry
0709 13:02:39.605849 (+107668us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b18004899a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:02:39.614318 (+  8469us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:39.614376 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.614454 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:40.324327 (+709873us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:40.547946 (+223619us) raft_consensus.cc:1685] finished
0709 13:02:40.547975 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:40.548146 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45657,"raft.queue_time_us":70078,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":8015,"spinlock_wait_cycles":121122944,"thread_start_us":8239,"threads_started":1,"wal-append.queue_time_us":152501}
I20260709 13:02:40.547827  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.444694 11837 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:40.555154  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.521907 11738 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.514544 11824 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.437879 11826 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.459232 11807 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:40.514628 11806 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:40.572156  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:40.489468 11805 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.008s	sys 0.020s
I20260709 13:02:40.447851 11578 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260709 13:02:40.576092  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.576845 11578 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.577142 11569 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.576152  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.547170 11632 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:40.451540 11785 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.118s
I20260709 13:02:40.579130  8580 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:40.580966  8732 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:40.579779  8724 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:40.581773  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 211187us
I20260709 13:02:40.586777  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:40.587749  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.521814 11808 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 13:02:40.450068 11768 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.124s
I20260709 13:02:40.590168  8625 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:40.565840 11837 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:40.577370 11578 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:40.599709  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.560518 11821 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 13:02:40.521683 11816 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:40.613596  8696 connection.cc:256] RPC call timeout handler was delayed by -1.1067193010000551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
I20260709 13:02:40.610558  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:40.641878  8696 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.679s (SENT)
I20260709 13:02:40.642190  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:40.642572  8696 connection.cc:256] RPC call timeout handler was delayed by -1.0446193010000684s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
W20260709 13:02:40.575045 11797 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.877s	user 0.003s	sys 0.025s
I20260709 13:02:40.555085 11813 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:40.643343  8696 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.716s (SENT)
I20260709 13:02:40.643924  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:40.645107  8696 leader_election.cc:336] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.826s (SENT)
W20260709 13:02:40.618077 11679 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.993s	user 0.007s	sys 0.006s
I20260709 13:02:40.647341 11679 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.648139  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:40.649092  8725 raft_consensus.cc:2410] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:40.649578 11679 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.645254 11813 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:40.650172 11844 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:02:40.650413  8577 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:40.651412 11692 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:40.651810 11692 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.652169 11692 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.652698  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 609) took 1141 ms. Trace:
I20260709 13:02:40.620455  8578 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:40.654318  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.620005 11806 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.625171  8479 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:40.655551  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.629554 11593 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.624076 11791 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.664453 11826 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:40.665441 11833 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 13:02:40.665609 11811 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:40.630909 11768 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:40.632593  8582 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:40.634238 11590 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.634840 11617 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.645565 11579 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.686411 11590 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:40.686281  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.686970 11590 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.645746  8576 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 3: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:40.577697 11569 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.607481 11632 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:40.643399 11797 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.952s	user 0.003s	sys 0.026s
I20260709 13:02:40.629313 11731 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.643307  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.653095  8577 rpcz_store.cc:276] 0709 13:02:39.510976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.534775 (+ 23799us) service_pool.cc:224] Handling call
0709 13:02:39.535207 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:40.650918 (+1115711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.652466 (+  1548us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540030b954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:40.652632 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2564042112}
I20260709 13:02:40.663123 11593 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:40.443837 11804 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.669687  8581 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:40.674978  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:40.567461 11822 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:40.680291  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 141716us
I20260709 13:02:40.725199  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:40.726542  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:40.683496 11848 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:40.579023 11824 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.684525  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.735970  8578 raft_consensus.cc:2410] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:40.685217 11739 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.686913 11579 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.646754 11604 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:40.688205  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.643267 11816 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:40.697985  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 52899us
W20260709 13:02:40.705809  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 194749us
I20260709 13:02:40.690814  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:40.832830  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:40.675671 11768 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.713449  8583 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5. Re-sending same reply.
I20260709 13:02:40.718551  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.719671  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:40.839457  8577 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:40.727517  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.728335  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:40.730010 11840 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.204s	user 0.004s	sys 0.000s
I20260709 13:02:40.730473  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:40.730669 11731 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:40.731369  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:40.731889 11704 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.731982  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:40.688884 11820 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.757592  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:40.714071 11593 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.865139 11593 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:40.810931  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 184204us
I20260709 13:02:40.815052  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.727038 11807 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 13:02:40.733718 11785 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:40.730469 11632 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:40.817749  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:40.814180  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 172009us
W20260709 13:02:40.825007  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.823670  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 98539.9us
W20260709 13:02:40.879951  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.881709  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.882025  9327 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 13:02:40.881081  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:40.826049  8582 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:40.883249  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:40.733777 11793 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 13:02:40.730469 11837 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:40.841466  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.842409  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.845734  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.854975  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:40.859728  9325 raft_consensus.cc:2376] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 6. Re-sending same reply.
I20260709 13:02:40.865712  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.866400  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.896271  8581 raft_consensus.cc:2376] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5. Re-sending same reply.
W20260709 13:02:40.866709  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 176011us
W20260709 13:02:40.898622  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.898490  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:40.899617  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.900167  8622 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.537s (SENT)
I20260709 13:02:40.900677  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:02:40.868185  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.868341  8696 leader_election.cc:336] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.617s (SENT)
I20260709 13:02:40.862768  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:40.904057  8696 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.683s (SENT)
I20260709 13:02:40.904402  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:40.830562 11786 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.680s	user 0.010s	sys 0.010s
I20260709 13:02:40.845615 11768 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.884114  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:40.896337  8580 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:40.905675  9326 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:40.906343  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:40.885612  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52825.4us
I20260709 13:02:40.907510  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.908221  8580 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:40.909101  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.890095  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:40.910256  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.904506  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:40.911644  8580 raft_consensus.cc:2410] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:40.911701  8575 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:40.914480  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.915694  8580 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:40.917080  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.917481 11768 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:40.919109  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.919806  8580 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:40.921792 11768 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:40.930141  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.823172 11848 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:40.931463  8580 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:40.867368 11617 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:40.940336  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.861541 11604 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.944389  9327 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:40.871622 11851 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:40.936707  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.943957  8579 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:02:40.948298  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46150.7us
I20260709 13:02:40.893854 11558 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:40.937633  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 82775.2us
I20260709 13:02:40.939528  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:40.951179  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51049.2us
I20260709 13:02:40.951342  8582 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:02:40.933136  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56127.8us
I20260709 13:02:40.840060 11805 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 13:02:40.874212 11623 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260709 13:02:40.937147  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:40.950381  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60408.7us
I20260709 13:02:40.948611  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.960227  8580 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:40.978193  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:40.883097 11799 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 13:02:40.981204  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:40.985757  8580 raft_consensus.cc:2376] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6. Re-sending same reply.
I20260709 13:02:40.947435 11851 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.984790  8732 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:40.988494  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:40.949512 11739 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:40.958726 11786 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:40.988494  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.991158  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:40.991446  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:41.005738  8651 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:41.000109  8725 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:40.957828 11590 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:40.957304 11842 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:40.989702  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 85659.4us
W20260709 13:02:41.002934  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.003872  8579 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
W20260709 13:02:40.988857  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 269205us
W20260709 13:02:41.011229  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.012403  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:41.013195  8650 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 13:02:41.013832  8696 leader_election.cc:336] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.830s (SENT)
I20260709 13:02:41.013967  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:41.014706  8581 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:41.014866  8696 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.899s (SENT)
I20260709 13:02:40.983549  9326 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:41.016716  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63966.3us
I20260709 13:02:40.955963 11808 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 13:02:41.020342  8577 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
W20260709 13:02:41.021170  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.113s (SENT)
I20260709 13:02:41.021584 11805 consensus_queue.cc:1243] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:40.958901 11558 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:41.021684 11857 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:02:41.014442 11623 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 13:02:41.038015  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:41.023078 11579 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.024197 11865 raft_consensus.cc:2764] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:02:40.988595 11569 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:40.950148 11604 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.015708 11786 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:41.028582  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:41.028986  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:41.025734  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:41.062088  8654 raft_consensus.cc:2368] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
I20260709 13:02:41.062223  8581 raft_consensus.cc:2410] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:41.063038  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:41.063771  8581 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:41.064904  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:41.034291  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:41.065775  8584 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
W20260709 13:02:41.035573  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 79609.3us
I20260709 13:02:40.998355 11851 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:41.012125 11739 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:41.005349 11798 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:41.036705  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
W20260709 13:02:41.032441  8548 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.538s (SENT)
I20260709 13:02:41.038938  8577 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:41.039444  8657 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:41.041115  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.043255  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:41.077344  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44944.1us
I20260709 13:02:41.072077  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:41.028204  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:41.057466  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.080470  8659 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:41.081770  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60687us
I20260709 13:02:41.036968 11590 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:41.075793  8584 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:02:41.077484  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:41.095712  8698 leader_election.cc:400] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
I20260709 13:02:41.096328  8698 leader_election.cc:403] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:02:41.038575 11858 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:41.083179  8579 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:41.073267  8656 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:41.036352 11852 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:41.039731 11861 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:41.037091 11822 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:41.082461  8696 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.172s (SENT)
I20260709 13:02:41.081933  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:41.116106  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.116145  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:41.117118  8654 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:41.116487  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98088.2us
I20260709 13:02:41.119819  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:41.095947 11865 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:41.097637 11805 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:41.099515 11824 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.018721 11848 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.115936 11806 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:41.109863 11861 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:41.155517 11578 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.130694 11590 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:41.103129 11798 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:41.213897  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 131456us
I20260709 13:02:41.110680 11871 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260709 13:02:41.139684 11820 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:41.221190  8696 leader_election.cc:336] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.186s (SENT)
I20260709 13:02:41.221899  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:41.144997 11579 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:41.231496  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:41.231992  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:41.213714 11855 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
I20260709 13:02:41.232965 11876 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:41.260456 11862 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
I20260709 13:02:41.261439 11881 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:41.262066 11881 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.264220 11881 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.240983 11864 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:41.205015 11739 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.268028  8580 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:41.258842  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:41.225174 11837 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.271659  8580 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 13:02:41.282099  8696 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:41.282395  8580 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:41.285281  8580 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 13:02:41.289254  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2343) took 1163 ms. Trace:
I20260709 13:02:41.301608  8575 rpcz_store.cc:276] 0709 13:02:40.125933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.177275 (+ 51342us) service_pool.cc:224] Handling call
0709 13:02:41.289060 (+1111785us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b54003e7454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:41.289193 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1671639296}
I20260709 13:02:41.290217  8577 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:41.287410  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:41.269709 11863 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.257s	user 0.000s	sys 0.001s
I20260709 13:02:41.304105  8580 pending_rounds.cc:85] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 2
I20260709 13:02:41.333439  8580 pending_rounds.cc:107] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 13:02:41.333839  8580 raft_consensus.cc:2889] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:41.243211  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:41.256542 11617 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:41.169754 11786 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:41.333791  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 214045us
I20260709 13:02:41.344624  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:41.291641  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:41.348863  8580 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 13:02:41.349485  8580 pending_rounds.cc:85] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 3
I20260709 13:02:41.349920  8580 pending_rounds.cc:107] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 13:02:41.350219  8580 raft_consensus.cc:2889] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:41.296517 11864 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:02:41.356230  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.292371 11865 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:41.363077  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71514.3us
W20260709 13:02:41.366477  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 284561us
I20260709 13:02:41.367626  8651 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:41.369522  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:41.371520  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 289761us
I20260709 13:02:41.352267 11799 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.377600  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:41.382656  8659 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:41.384225  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:41.367084 11791 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.012s
I20260709 13:02:41.388294  8724 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:41.390359  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 294860us
W20260709 13:02:41.392431  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47927.5us
I20260709 13:02:41.396358  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:41.282166 11876 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.347901  9324 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:41.366839 11871 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:41.398865  8583 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:41.402715  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:41.409824  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:41.410734  8731 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:02:41.410948  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.413637  8655 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260709 13:02:41.375463 11604 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.421960  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:41.414270  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:41.417649  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.419608  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:41.414135  8655 pending_rounds.cc:85] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 6
I20260709 13:02:41.423816  8651 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:41.441521  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.442135  8726 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:41.441627  8655 pending_rounds.cc:107] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 13:02:41.444804  8655 raft_consensus.cc:2889] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:41.447991  8728 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
W20260709 13:02:41.449107  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.449628  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:41.392547 11805 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:41.330128 11861 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.395721 11799 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:41.450484  8726 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:41.570936  8650 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:41.483841  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.557991  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 178587us
I20260709 13:02:41.490600  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.490813  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:41.644874  8548 leader_election.cc:336] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.734s (SENT)
I20260709 13:02:41.646411  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:41.646523  8652 raft_consensus.cc:2368] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8.
I20260709 13:02:41.646736  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:41.554286  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.516640  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:41.463682  8731 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:41.613584  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:41.660055  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 257273us
I20260709 13:02:41.452104  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:41.414431 11579 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:41.665099  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:41.402311 11837 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:41.537467  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:41.559904  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:41.673867  8651 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:41.688271  8728 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:41.454046  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2357) took 1268 ms. Trace:
I20260709 13:02:41.689152  8576 rpcz_store.cc:276] 0709 13:02:40.185742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.418540 (+232798us) service_pool.cc:224] Handling call
0709 13:02:41.453850 (+1035310us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400320fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:41.453984 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309428608}
I20260709 13:02:41.693673  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:41.695154  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:41.413929 11558 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:41.489766 11809 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260709 13:02:41.702256  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 57411.7us
I20260709 13:02:41.716235  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:41.653776 11798 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.723732  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:41.726128  8548 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.032s (SENT)
I20260709 13:02:41.726770 11819 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:41.726851  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:41.726235 11859 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:41.728776 11859 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:41.729238 11859 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:41.729919 11798 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:41.730387 11876 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.730459 11798 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.731673 11875 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.732262 11875 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.733608 11783 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:41.734059 11783 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:41.734948 11875 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:41.735059 11783 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:41.736296 11783 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:41.736981 11797 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:41.737669 11904 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:02:41.738493 11884 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260709 13:02:41.739245 11904 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:41.735481 11903 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:02:41.742030 11888 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:02:41.560382 11890 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:41.747014  8652 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [4.4-7.5]
I20260709 13:02:41.743647  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 813) took 1021 ms. Trace:
I20260709 13:02:41.755971  8651 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:41.661715 11898 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:41.446051  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:02:41.771720  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 106681us
I20260709 13:02:41.790161  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:41.791735  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:41.794345  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70726.9us
I20260709 13:02:41.799522  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:41.451622 11604 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.313s
I20260709 13:02:41.710428 11871 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:41.755223  9330 rpcz_store.cc:276] 0709 13:02:40.722299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.725728 (+  3429us) service_pool.cc:224] Handling call
0709 13:02:40.729735 (+  4007us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.818896 (+ 89161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.818926 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:40.830604 (+ 11678us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.150351 (+319747us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b1800419680. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:02:41.308182 (+157831us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:41.339482 (+ 31300us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.579246 (+239764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.579384 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.743312 (+163928us) raft_consensus.cc:1685] finished
0709 13:02:41.743436 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.743581 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58674,"prepare.queue_time_us":31318,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":8635,"raft.queue_time_us":499740,"raft.run_cpu_time_us":445,"raft.run_wall_time_us":91957,"spinlock_wait_cycles":2606976,"thread_start_us":462037,"threads_started":3,"wal-append.queue_time_us":591796}
I20260709 13:02:41.744558 11874 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:41.831533  8658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.015s	sys 0.027s
W20260709 13:02:41.915195  8579 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.002s	sys 0.017s
I20260709 13:02:41.797113  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:41.758124 11822 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:41.756014 11579 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:41.936700  8579 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.003s	sys 0.018s
I20260709 13:02:41.917912  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:41.945216  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.956198  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.919641  8548 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.194s (SENT)
W20260709 13:02:41.947549  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 150519us
W20260709 13:02:41.801061  8466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 361089us
W20260709 13:02:41.833544 11739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.003s	sys 0.009s
I20260709 13:02:41.801685 11824 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.962392  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:41.959360  8728 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
W20260709 13:02:41.834985 11623 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.001s	sys 0.005s
W20260709 13:02:41.918949 11738 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.345s	user 0.013s	sys 0.000s
W20260709 13:02:41.963099  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 443194us
W20260709 13:02:41.936700  8658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.028s	sys 0.042s
I20260709 13:02:41.802632 11797 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:41.974854  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57242us
W20260709 13:02:41.954509 11877 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.593s	user 0.002s	sys 0.000s
I20260709 13:02:41.787153 11791 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.921824 11865 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:42.003747  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:41.948949 11822 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.014146  8580 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:42.017684  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:41.967772 11739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.007s	sys 0.015s
I20260709 13:02:42.010120  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:41.953289 11837 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:41.985371  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65734.2us
I20260709 13:02:41.979167  8655 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20260709 13:02:41.972486 11623 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.001s	sys 0.018s
I20260709 13:02:41.966421 11848 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:42.019243 11865 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:42.041810 11865 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:41.765668 11880 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.318s
W20260709 13:02:42.054608  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.007301  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:42.021087 11822 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:42.030360 11738 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.009150 11808 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:42.090042  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82865.5us
I20260709 13:02:42.002133 11768 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260709 13:02:41.978868 11876 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.067s
I20260709 13:02:42.033111 11579 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:42.082463 11856 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 13:02:42.101256 11883 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:42.101711 11878 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:41.944265 11805 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.096932  8726 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:41.981455 11819 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260709 13:02:41.972075 11824 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.016198 11886 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.126s
I20260709 13:02:42.197697 11798 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:42.137173 11916 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.137045 11808 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.800554 11898 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:41.957700 11871 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.161348  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:42.120669  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:42.162950  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:42.137619 11896 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.541s	user 0.001s	sys 0.000s
I20260709 13:02:42.177034  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:42.131400 11593 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.265s	user 0.011s	sys 0.004s
W20260709 13:02:42.118669 11851 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.033s	user 0.013s	sys 0.000s
I20260709 13:02:42.204241  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.109862 11911 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:42.141831 11692 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.489s	user 0.000s	sys 0.015s
I20260709 13:02:42.190640  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:42.193472  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:42.219750 11808 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:42.227558 11590 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.009s	user 0.010s	sys 0.003s
I20260709 13:02:42.234515 11899 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.161110 11738 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.234979 11590 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 3. Current term is 4.
I20260709 13:02:42.235215 11593 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.203353 11891 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.237694 11851 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.238168 11891 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.239835 11851 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:42.240291 11891 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:42.240869  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 79601.4us
W20260709 13:02:42.197202 11731 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.338s	user 0.007s	sys 0.015s
I20260709 13:02:42.242324  9328 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:42.101888 11793 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:42.023913 11797 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.242761  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:42.211160 11895 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.267s	user 0.001s	sys 0.000s
I20260709 13:02:42.206949 11878 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:42.244621  9328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 812) took 1522 ms (client timeout 1734 ms). Trace:
I20260709 13:02:42.244813 11797 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:42.245100  9328 rpcz_store.cc:269] 0709 13:02:40.722030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.726229 (+  4199us) service_pool.cc:224] Handling call
0709 13:02:42.244431 (+1518202us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54006900d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:42.244547 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3220646784}
I20260709 13:02:42.245983 11797 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:42.246580 11623 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:42.023061 11791 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.246925 11797 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:42.247421  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.248441 11918 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 13:02:42.249274  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1794) took 1543 ms. Trace:
I20260709 13:02:42.249498  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.251166  9328 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:42.252377  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.179546 11579 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:42.137949 11908 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:42.254046 11579 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:42.254119 11908 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:42.254700 11908 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:42.254798  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 615) took 1865 ms. Trace:
I20260709 13:02:42.255275  8581 rpcz_store.cc:276] 0709 13:02:40.388884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.073982 (+685098us) service_pool.cc:224] Handling call
0709 13:02:41.074221 (+   239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.239996 (+165775us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.240043 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:41.240100 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:41.240135 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.254563 (+1014428us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54003449d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:42.254614 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.254740 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":380668672}
I20260709 13:02:42.256598  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:42.218480 11906 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
I20260709 13:02:42.219271 11824 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:42.234056 11692 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.234722 11569 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:42.210403 11915 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:02:42.243081 11793 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:42.248719 11791 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:42.189142 11858 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.073s	user 0.012s	sys 0.000s
I20260709 13:02:42.189925 11799 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:42.259538  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:42.260371 11807 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:42.264032  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.261288  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1793) took 1555 ms. Trace:
I20260709 13:02:42.289969  8659 raft_consensus.cc:2368] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6.
I20260709 13:02:42.217671 11916 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:42.293310  8724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.000s	sys 0.025s
I20260709 13:02:42.273058  8732 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 4.
I20260709 13:02:42.195627 11859 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.269574  8582 rpcz_store.cc:276] 0709 13:02:40.706354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.161026 (+454672us) service_pool.cc:224] Handling call
0709 13:02:41.169174 (+  8148us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:41.215937 (+ 46763us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.215968 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:41.227608 (+ 11640us) log.cc:844] Serialized 52 byte log entry
0709 13:02:41.376377 (+148769us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:41.376416 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.376480 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.249057 (+872577us) raft_consensus.cc:1685] finished
0709 13:02:42.249080 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.249208 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4074,"raft.queue_time_us":21512,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":32844,"spinlock_wait_cycles":8448,"thread_start_us":144726,"threads_started":1,"wal-append.queue_time_us":479968}
I20260709 13:02:42.283299  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:42.284952  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.247398 11797 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:42.248756  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:42.257537 11880 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:42.234380 11919 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 13:02:42.234316  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 113731us
W20260709 13:02:42.294659  8724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.000s	sys 0.025s
I20260709 13:02:42.295876  8581 raft_consensus.cc:2410] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:02:42.297472  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:42.286154 11632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.000s	sys 0.021s
W20260709 13:02:42.304563  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.290175  8579 rpcz_store.cc:276] 0709 13:02:40.705929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.100343 (+394414us) service_pool.cc:224] Handling call
0709 13:02:41.123024 (+ 22681us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.123527 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.123594 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.141336 (+ 17742us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.954691 (+813355us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:41.954833 (+   142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.955058 (+   225us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.261050 (+305992us) raft_consensus.cc:1685] finished
0709 13:02:42.261069 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.261215 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8170,"raft.queue_time_us":42500,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":42988,"thread_start_us":730469,"threads_started":1,"wal-append.queue_time_us":1082486}
I20260709 13:02:42.277842 11623 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:42.316466  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:42.319092  8578 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:42.292855 11891 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.314713  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:42.289714 11858 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:42.289949 11807 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:42.325793  8581 pending_rounds.cc:85] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 6
I20260709 13:02:42.329030  8581 pending_rounds.cc:107] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:02:42.329363  8581 raft_consensus.cc:2889] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:42.313777 11632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.915s	user 0.000s	sys 0.022s
I20260709 13:02:42.296952 11808 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.334756  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:42.329465  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:42.336649  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:42.330921  8697 leader_election.cc:336] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.614s (SENT)
I20260709 13:02:42.235216 11899 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.333621  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:42.314313 11804 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:42.313745 11904 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:42.341368  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:42.341879  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 818) took 1618 ms. Trace:
I20260709 13:02:42.335605  8579 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:42.320502 11593 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:42.276875 11911 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.355360 11873 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:42.355770 11692 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:42.343483 11843 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:42.357933  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:42.759799 11843 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.760335 11843 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:42.358891 11886 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:42.761608 11886 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:42.359171 11783 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.000s
I20260709 13:02:42.362319 11837 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:42.362498 11865 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.363301 11822 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.367565  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:42.344833  9322 rpcz_store.cc:276] 0709 13:02:40.723660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.895317 (+171657us) service_pool.cc:224] Handling call
0709 13:02:40.951945 (+ 56628us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.351321 (+399376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.351355 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.351909 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.448834 (+ 96925us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:41.449741 (+   907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.449820 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.341608 (+891788us) raft_consensus.cc:1685] finished
0709 13:02:42.341637 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.341804 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1466,"raft.queue_time_us":65063,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":467826,"threads_started":2,"wal-append.queue_time_us":385176}
I20260709 13:02:42.320967 11878 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:42.310775 11901 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:02:42.330858 11793 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:42.351087 11797 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.516464 11911 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:42.525780  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:42.426870 11807 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:42.408069  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 247713us
W20260709 13:02:42.467844 11933 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 13:02:42.518407 11859 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:42.520128 11871 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:42.507997  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2379) took 1859 ms. Trace:
I20260709 13:02:42.386917  8578 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:42.612047  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 298201us
I20260709 13:02:42.870693  8577 rpcz_store.cc:276] 0709 13:02:40.648841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.084069 (+435228us) service_pool.cc:224] Handling call
0709 13:02:41.289815 (+205746us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.303759 (+ 13944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.507689 (+1203930us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54003e7450. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:42.507907 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":472408960}
I20260709 13:02:42.871735  9296 leader_election.cc:400] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8.
I20260709 13:02:42.872136  9296 leader_election.cc:403] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:02:42.318004 11880 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:42.449270  8731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.012s	sys 0.019s
I20260709 13:02:42.332095 11893 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20260709 13:02:42.337874 11876 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:42.458989 11916 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:42.617942 11579 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.513087 11874 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:42.485280 11908 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.897356  8733 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
W20260709 13:02:42.645345 11925 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 13:02:42.725997  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 225217us
I20260709 13:02:42.500299  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:42.521010  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:42.511261 11679 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:42.735424  9330 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:42.370589 11785 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:42.344026 11808 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:42.752617  8697 leader_election.cc:336] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.872s (SENT)
W20260709 13:02:42.760216  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 402343us
I20260709 13:02:42.333102 11891 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:42.839689  8583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1816) took 2120 ms (client timeout 1636 ms). Trace:
I20260709 13:02:42.822714  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:42.827625  8729 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:42.823280  8575 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:42.842096  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 474529us
W20260709 13:02:42.820981  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.719691 11883 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:42.816501 11852 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:42.859989  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:42.907804  8548 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.358s (SENT)
I20260709 13:02:42.911870  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:42.912389  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:42.831972 11868 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.607s	user 0.000s	sys 0.014s
I20260709 13:02:42.762189 11886 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:42.850587 11864 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:42.878891  8731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.961s	user 0.028s	sys 0.043s
I20260709 13:02:42.812215 11861 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.882328 11938 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.883641  9322 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:42.883688  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.884423  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.884891  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.880538 11876 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.851500 11899 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:42.880939  8479 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 4, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 4 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:42.836289 11812 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 13:02:42.850992 11901 consensus_queue.cc:1243] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:42.853590 11873 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:42.902781  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:42.905126 11916 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:42.904817  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.906447  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.971827  9330 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:42.906587  8583 rpcz_store.cc:269] 0709 13:02:40.719597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.839366 (+119769us) service_pool.cc:224] Handling call
0709 13:02:42.839473 (+2000107us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b540017a054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:42.839609 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1151730560}
I20260709 13:02:42.924216 11864 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.896962 11874 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:42.905339  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:42.920715  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:42.903564 11679 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:42.966565 11876 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:42.943840 11886 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.920476 11816 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:43.011772 11940 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
W20260709 13:02:42.950134 11949 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.114s	user 0.002s	sys 0.000s
I20260709 13:02:43.015429  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2403) took 2064 ms. Trace:
I20260709 13:02:42.930402 11604 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 13:02:42.919045 11852 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:42.982954 11593 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:42.907231 11798 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:42.970949  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 68276.8us
I20260709 13:02:42.899866  8655 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:42.906970  8582 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:42.973632  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:42.924834 11878 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.029655  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:43.030198 11878 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.958635 11883 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.984963  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:42.962155 11898 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:43.033362  8577 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:43.033272 11883 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:42.995952  8651 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:42.939772  8652 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:42.996466  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2414) took 1952 ms. Trace:
I20260709 13:02:43.006160 11873 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.008201  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 815) took 2285 ms. Trace:
I20260709 13:02:43.039248  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:42.998991  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:43.015991  8657 rpcz_store.cc:276] 0709 13:02:40.951463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.951609 (+   146us) service_pool.cc:224] Handling call
0709 13:02:40.977901 (+ 26292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.084544 (+106643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.015134 (+1930590us) spinlock_profiling.cc:243] Waited 61.2 ms on lock 0x7b5400380254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:43.015356 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140716288}
I20260709 13:02:42.954676 11865 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:43.042235  8575 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:43.018805  8729 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:43.020624  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 114748us
W20260709 13:02:43.021698  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.044943  8582 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:42.909502 11738 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.025954  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:42.984712 11864 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:43.045691  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:43.023808  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.046380  8655 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 13:02:43.039247  9323 rpcz_store.cc:276] 0709 13:02:40.722895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.728733 (+  5838us) service_pool.cc:224] Handling call
0709 13:02:40.729048 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.729363 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.729387 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:40.737196 (+  7809us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.022156 (+284960us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:41.022193 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.022272 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.831349 (+1809077us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:43.007864 (+176515us) raft_consensus.cc:1685] finished
0709 13:02:43.007888 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.008139 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":125792,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":14727,"spinlock_wait_cycles":6660864,"thread_start_us":151693,"threads_started":1,"wal-append.queue_time_us":251661}
I20260709 13:02:43.049022  8577 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:43.036948  8658 rpcz_store.cc:276] 0709 13:02:41.043633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.043766 (+   133us) service_pool.cc:224] Handling call
0709 13:02:41.076242 (+ 32476us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:41.167177 (+ 90935us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.167213 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:41.167985 (+   772us) log.cc:844] Serialized 78 byte log entry
0709 13:02:41.969795 (+801810us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:42.519183 (+549388us) spinlock_profiling.cc:243] Waited 8.66 ms on lock 0x7b18002e14a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:02:42.858729 (+339546us) log.cc:844] Serialized 14 byte log entry
0709 13:02:42.980615 (+121886us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.980711 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:42.980780 (+    69us) raft_consensus.cc:1685] finished
0709 13:02:42.996254 (+ 15474us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.996406 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48740,"prepare.queue_time_us":672475,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":147794,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":22290,"spinlock_wait_cycles":19912064,"thread_start_us":1223185,"threads_started":4,"wal-append.queue_time_us":751108}
I20260709 13:02:43.024662 11811 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.985462 11916 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.040323 11783 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.081871  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:43.101887  8579 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:43.085875  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 479) took 1644 ms. Trace:
I20260709 13:02:43.047283 11593 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:43.058925  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.059460  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:43.108870  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:43.058240  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.110221  8582 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:43.096747  8575 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:43.081049  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:43.084707  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:43.126420  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 67078.2us
I20260709 13:02:43.060540 11898 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:43.132863  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:43.134513  8726 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:02:43.100392 11816 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.087682 11899 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:43.098855 11875 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:43.151175  8725 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 13:02:43.147841  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:43.099009 11951 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.039708 11798 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:43.099891 11822 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.158751  8725 pending_rounds.cc:85] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:43.196760  8725 pending_rounds.cc:107] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 13:02:43.159567  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.161084  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.166209  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:43.200551  8728 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:43.200467  8730 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
W20260709 13:02:43.153659  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 401036us
I20260709 13:02:43.174109  8726 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:43.208154  8726 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:43.099308 11811 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.212039  8725 raft_consensus.cc:2889] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:43.215129  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 189004us
I20260709 13:02:43.209293  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.209604  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:43.150728 11738 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:43.218308  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:43.223258  8730 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:43.237293  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1967) took 1665 ms. Trace:
W20260709 13:02:43.257066  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47566.6us
I20260709 13:02:43.217763  8726 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:43.151046 11954 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:43.258054  8577 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:43.123147 11783 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:43.153342 11880 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.259559  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.261992  8725 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:43.260663  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.258090  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:43.263252  8725 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:43.265378  8725 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:02:43.155550 11816 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.274950  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.256955  8731 rpcz_store.cc:276] 0709 13:02:41.572213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.654877 (+ 82664us) service_pool.cc:224] Handling call
0709 13:02:41.655330 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.655649 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:41.917597 (+261948us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.235386 (+1317789us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.236931 (+  1545us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:43.236982 (+    51us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:43.237033 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.237102 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.237231 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":213,"prepare.queue_time_us":596066,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":43,"raft.queue_time_us":160548,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":12808,"thread_start_us":1466154,"threads_started":4,"wal-append.queue_time_us":571787}
I20260709 13:02:43.211408 11579 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:43.157770 11899 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.279803 11738 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:43.154826 11898 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:43.273761 11843 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.003s
I20260709 13:02:43.129880 11679 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.257467 11822 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:43.296283  8728 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:43.297930  8731 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:02:43.259951 11808 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:43.265260 11908 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:43.274974 11954 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:43.344780 11908 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:43.207710 11951 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:43.345301 11908 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:43.345088 11843 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.346366 11908 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.346875 11951 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:43.280711 11952 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.323902  8731 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:43.263700 11593 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:43.328977 11938 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8.
I20260709 13:02:43.327577 11798 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:43.328881 11816 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.329247 11811 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:43.330384 11916 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.331707 11953 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:43.334806  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.334589 11852 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.336215  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.339627  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:43.340015 11558 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:43.343340 11899 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:43.355019  8579 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:43.325764 11812 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 13:02:43.313727 11579 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:43.446200  8579 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:43.280391 11891 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:43.360060  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:43.320933 11939 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.324s	user 0.001s	sys 0.000s
W20260709 13:02:43.365621  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 220757us
I20260709 13:02:43.366825  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.458134  8584 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:43.345127 11954 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.347927 11843 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.363483 11822 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:43.438441  8575 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:43.441043  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.437503 11852 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:43.454365  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:43.497565  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:43.498271  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.466192 11959 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:43.467301  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 472) took 2063 ms (client timeout 1866 ms). Trace:
W20260709 13:02:43.499814  8659 rpcz_store.cc:269] 0709 13:02:41.410194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.410637 (+   443us) service_pool.cc:224] Handling call
0709 13:02:43.467038 (+2056401us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b54000596d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:43.467206 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2019105536}
W20260709 13:02:43.468281  8652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2290) took 1504 ms (client timeout 1972 ms). Trace:
W20260709 13:02:43.469589 11932 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.012s	user 0.007s	sys 0.005s
I20260709 13:02:43.470420  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:43.470516 11849 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:43.470446 11783 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:43.471827  9329 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:43.471992 11731 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:43.475493 11864 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:43.476452 11861 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.000s
I20260709 13:02:43.489801  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:43.492286  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:43.500909  8652 rpcz_store.cc:269] 0709 13:02:41.964113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.964261 (+   148us) service_pool.cc:224] Handling call
0709 13:02:43.468095 (+1503834us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b5400424854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:43.468209 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1904208640}
W20260709 13:02:43.471928 11931 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.013s	user 0.003s	sys 0.007s
I20260709 13:02:43.502031  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.498871  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:43.519665  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:43.458879 11811 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:43.506955  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 288690us
I20260709 13:02:43.528497  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:43.503295 11886 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:43.510299  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 385495us
I20260709 13:02:43.525854  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:43.558531  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:43.585503  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57064.6us
I20260709 13:02:43.586822  8696 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:43.592871  8575 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:43.595242  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 622) took 1843 ms. Trace:
I20260709 13:02:43.598075 11811 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.599629 11783 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:43.599808  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:43.754467  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 154714us
I20260709 13:02:43.755173 11826 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:43.602241 11966 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:02:43.367658 11952 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.511567 11899 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:43.606370  8729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1966) took 2071 ms (client timeout 1960 ms). Trace:
W20260709 13:02:43.772152  8729 rpcz_store.cc:269] 0709 13:02:41.535084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.535216 (+   132us) service_pool.cc:224] Handling call
0709 13:02:43.606091 (+2070875us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400064d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:43.606268 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2678074624}
I20260709 13:02:43.505505 11951 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:43.681967 11558 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:43.758351 11783 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:43.787696 11783 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:43.614140  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.788811  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 477) took 2371 ms. Trace:
I20260709 13:02:43.516251 11731 consensus_queue.cc:1243] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:43.684033  8726 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:43.729455  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 142743us
I20260709 13:02:43.726398  8576 rpcz_store.cc:276] 0709 13:02:41.751786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.754238 (+  2452us) service_pool.cc:224] Handling call
0709 13:02:41.797871 (+ 43633us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.961565 (+163694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.961608 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:42.003821 (+ 42213us) log.cc:844] Serialized 26 byte log entry
0709 13:02:42.305313 (+301492us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:42.305380 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.305501 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.471275 (+1165774us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:43.594823 (+123548us) raft_consensus.cc:1685] finished
0709 13:02:43.594853 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.595148 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115164,"raft.queue_time_us":218774,"raft.run_cpu_time_us":2381,"raft.run_wall_time_us":76753,"spinlock_wait_cycles":7983488,"thread_start_us":247837,"threads_started":1,"wal-append.queue_time_us":430091}
W20260709 13:02:43.758325  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.008s	sys 0.037s
W20260709 13:02:43.798601  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.008s	sys 0.037s
I20260709 13:02:43.511188 11891 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.173s
I20260709 13:02:43.536590 11901 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.602069 11953 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:43.805217  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.491009 11738 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.806180  8729 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:43.807144  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.807790  8729 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:43.808725  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.536568 11969 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:43.105654  8653 rpcz_store.cc:276] 0709 13:02:41.441502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.442251 (+   749us) service_pool.cc:224] Handling call
0709 13:02:41.516285 (+ 74034us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:41.553258 (+ 36973us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.553293 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:41.709462 (+156169us) log.cc:844] Serialized 52 byte log entry
0709 13:02:42.163756 (+454294us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:42.909925 (+746169us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.056530 (+146605us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.085304 (+ 28774us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.085587 (+   283us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.085623 (+    36us) raft_consensus.cc:1685] finished
0709 13:02:43.085633 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.085802 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":458226,"prepare.queue_time_us":728125,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":1966,"raft.queue_time_us":82175,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":24157,"thread_start_us":765050,"threads_started":3,"wal-append.queue_time_us":539584}
I20260709 13:02:43.809666 11969 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:43.811223 11971 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:02:43.811076 11969 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:43.813944  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:43.814826  8659 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
W20260709 13:02:43.815871  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1971) took 1688 ms (client timeout 1938 ms). Trace:
I20260709 13:02:43.816071 11901 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:43.816548  8733 rpcz_store.cc:269] 0709 13:02:42.127115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.127524 (+   409us) service_pool.cc:224] Handling call
0709 13:02:43.815703 (+1688179us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b540042fed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:43.815807 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1456460672}
I20260709 13:02:43.818079  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.818470 11891 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:43.819075 11891 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:43.815914  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:43.779798  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.595259  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:43.781181 11899 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:43.513356 11849 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:43.785845 11864 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.506799 11959 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.751180 11579 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:43.800979  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:43.832553  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.020s	sys 0.068s
W20260709 13:02:43.832929  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.020s	sys 0.068s
I20260709 13:02:43.832707  8652 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:43.782368 11951 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:43.804402  8655 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 13:02:43.804352  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1960) took 2406 ms. Trace:
I20260709 13:02:43.799067  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:43.780519  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.821511  8728 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:43.823835  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.827261  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:43.828910  8575 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:43.789191  8654 rpcz_store.cc:276] 0709 13:02:41.417579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.417816 (+   237us) service_pool.cc:224] Handling call
0709 13:02:41.418233 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.516446 (+ 98213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.516493 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.643822 (+127329us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.918974 (+275152us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:42.311558 (+392584us) log.cc:844] Serialized 14 byte log entry
0709 13:02:42.311873 (+   315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.771833 (+1459960us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b540005a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:43.771899 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.771960 (+    61us) raft_consensus.cc:1685] finished
0709 13:02:43.788620 (+ 16660us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.788758 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":177568,"prepare.queue_time_us":629875,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":60725,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":32491,"spinlock_wait_cycles":91944320,"thread_start_us":455853,"threads_started":2,"wal-append.queue_time_us":501139,"wal-append.run_cpu_time_us":2614,"wal-append.run_wall_time_us":1196804}
W20260709 13:02:43.803515 11874 consensus_meta.cc:287] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.793s	user 0.012s	sys 0.000s
W20260709 13:02:44.072254  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 1955) took 2695 ms (client timeout 1734 ms). Trace:
I20260709 13:02:44.120348  8576 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:44.120837  8655 pending_rounds.cc:85] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 3
W20260709 13:02:43.965894  9296 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.889s	user 0.019s	sys 0.014s
I20260709 13:02:43.834322  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:44.121654  9296 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.045s	user 0.022s	sys 0.014s
I20260709 13:02:43.866590  8582 raft_consensus.cc:2410] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:44.122857  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:44.007433  8549 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.725s (SENT) [suppressed 42 similar messages]
W20260709 13:02:44.124312  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.862576  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:43.931396  8726 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:02:44.120401  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:44.022851  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.009s	sys 0.016s
W20260709 13:02:43.966100  8583 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.017s	sys 0.011s
W20260709 13:02:44.119949  8732 rpcz_store.cc:269] 0709 13:02:41.376694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.377231 (+   537us) service_pool.cc:224] Handling call
0709 13:02:44.071955 (+2694724us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b5400436054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:44.072126 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1955582720}
W20260709 13:02:44.122047  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.04565e+06us
I20260709 13:02:43.822438 11865 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.362s
I20260709 13:02:44.235579 11604 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.235911 11864 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.244688 11604 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:44.120986  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.245728 11864 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.246234  8657 raft_consensus.cc:2376] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 7. Re-sending same reply.
I20260709 13:02:44.247589  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:44.212853  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2394) took 1231 ms. Trace:
I20260709 13:02:44.248148  8655 pending_rounds.cc:107] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 6.4
W20260709 13:02:44.243274  9329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 824) took 3004 ms (client timeout 1794 ms). Trace:
I20260709 13:02:44.248659  8655 raft_consensus.cc:2889] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:44.248672  9329 rpcz_store.cc:269] 0709 13:02:41.238775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.240698 (+  1923us) service_pool.cc:224] Handling call
0709 13:02:42.500088 (+1259390us) spinlock_profiling.cc:243] Waited 8.68 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:02:44.242894 (+1742806us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b5400816354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:44.243128 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1114989184}
I20260709 13:02:43.824817 11899 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:44.153842  8578 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:43.832743  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.258323  8659 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 4: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:44.110677 11704 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.248383  8577 rpcz_store.cc:276] 0709 13:02:42.981736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.141604 (+159868us) service_pool.cc:224] Handling call
0709 13:02:44.212623 (+1071019us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b54000f87d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:44.212789 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":133177728}
I20260709 13:02:44.119982  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:44.272032  8728 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:44.127776  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 328791us
I20260709 13:02:44.110337 11785 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:44.259509  8583 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.896s	user 0.021s	sys 0.015s
I20260709 13:02:44.175863  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:44.256690  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.010s	sys 0.021s
I20260709 13:02:44.251282  8658 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:44.299046  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.177100 11930 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
I20260709 13:02:44.250375  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.322614  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.324154  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.324863  8655 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:44.325564 11786 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.150s	user 0.007s	sys 0.045s
W20260709 13:02:44.326406 11890 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.001s	sys 0.001s
W20260709 13:02:44.326872 11890 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.001s	sys 0.001s
I20260709 13:02:44.328656  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:44.329689 11738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.000s	sys 0.005s
W20260709 13:02:44.330201 11738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.001s	sys 0.005s
W20260709 13:02:44.329075 11880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.012s	sys 0.012s
W20260709 13:02:44.330670 11880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.012s	sys 0.012s
I20260709 13:02:44.330935  8655 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:44.110781 11793 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:44.208509 11876 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.183s	user 0.008s	sys 0.004s
I20260709 13:02:43.815944 11811 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:44.257177  8724 rpcz_store.cc:276] 0709 13:02:41.398437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.399761 (+  1324us) service_pool.cc:224] Handling call
0709 13:02:41.400180 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.568386 (+168206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.568430 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.625142 (+ 56712us) log.cc:844] Serialized 26 byte log entry
0709 13:02:42.312488 (+687346us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:42.312548 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.312651 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.468951 (+1156300us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:43.791187 (+322236us) raft_consensus.cc:1685] finished
0709 13:02:43.791213 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.804264 (+ 13051us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4506,"raft.queue_time_us":248968,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":14284,"spinlock_wait_cycles":23088896,"thread_start_us":731366,"threads_started":2,"wal-append.queue_time_us":763072}
I20260709 13:02:44.334681  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:44.335741  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:44.076640 11875 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.919s	user 0.003s	sys 0.007s
I20260709 13:02:44.168064 11692 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.275760  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:44.358613  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82943.7us
I20260709 13:02:44.253983 11604 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:44.125079 11951 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:44.337639 11617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.029s	sys 0.037s
I20260709 13:02:44.338024  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:44.339295  8623 connection.cc:256] RPC call timeout handler was delayed by -1.652831185000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1574s for a response
I20260709 13:02:44.110895 11837 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.269352  8479 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 5, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 5 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:43.980921 11579 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.922888 11952 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:43.863903 11959 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:44.071039 11980 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.863903 11849 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:44.110941 11893 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:44.346865  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:44.354282  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 178515us
I20260709 13:02:44.207907 11901 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:44.229059 11898 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:44.356120  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:44.174561 11977 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:44.357828  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:44.336609 11786 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.176s	user 0.007s	sys 0.045s
I20260709 13:02:44.333575  8655 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:02:44.194134  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:44.338699 11864 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:44.367142 11972 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:44.124053 11874 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.407611 11874 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:44.407827  8728 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:44.409655  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:44.411733  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:44.412868  9324 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:02:44.359373 11867 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:44.415733 11867 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.317314 11785 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:44.337886 11938 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.422104 11938 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.316860 11704 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:44.426298  8654 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:44.428028 11704 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:44.429522 11977 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:44.430804 11977 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:44.338428 11865 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 13:02:44.435214 11865 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.435849 11865 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:43.954388 11891 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.076074 11783 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.444604 11783 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:44.446134  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.447017  8652 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:44.371490  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.372509  9328 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:44.466202  8624 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.005s	sys 0.036s
W20260709 13:02:44.466740  8624 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.005s	sys 0.036s
I20260709 13:02:44.388954  9326 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:44.388219  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:44.394517  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2392) took 1413 ms. Trace:
I20260709 13:02:44.392794  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.468127  8582 rpcz_store.cc:276] 0709 13:02:42.980984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.440520 (+459536us) service_pool.cc:224] Handling call
0709 13:02:44.394246 (+953726us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54001694d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:44.394440 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":852400512}
I20260709 13:02:44.469094  8653 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:44.391412  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.424611 11938 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:44.382261  8623 connection.cc:256] RPC call timeout handler was delayed by -1.4800311850000298s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260709 13:02:44.475523  8623 connection.cc:256] RPC call timeout handler was delayed by -1.4548311850000459s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1794s for a response
W20260709 13:02:44.476027  8623 connection.cc:256] RPC call timeout handler was delayed by -1.4060096790000216s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
W20260709 13:02:44.476454  8623 connection.cc:256] RPC call timeout handler was delayed by -1.2977528069999948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
W20260709 13:02:44.476881  8623 connection.cc:256] RPC call timeout handler was delayed by -1.2505311850000567s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2021s for a response
I20260709 13:02:44.389696  9327 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:44.387710  8658 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:44.371932  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:44.479422  9323 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:44.462745  8659 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:44.479765  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 91646.9us
I20260709 13:02:44.481483  8659 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 13:02:44.429204 11604 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:44.467134  8726 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:44.470589  8650 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:44.482960  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:44.461503  8725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.020s	sys 0.038s
I20260709 13:02:44.492417  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:44.505520  8584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s	user 0.015s	sys 0.028s
W20260709 13:02:44.544550  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 66360.9us
W20260709 13:02:44.543447  8730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.031s	user 0.000s	sys 0.037s
W20260709 13:02:44.545603  8725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.026s	sys 0.040s
W20260709 13:02:44.545920  8730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.000s	sys 0.037s
I20260709 13:02:44.546182  8623 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:44.546689  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2294) took 1281 ms. Trace:
I20260709 13:02:44.547276  8725 rpcz_store.cc:276] 0709 13:02:43.265608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.280292 (+ 14684us) service_pool.cc:224] Handling call
0709 13:02:43.280582 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:43.280767 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:43.603297 (+322530us) log.cc:844] Serialized 14 byte log entry
0709 13:02:44.546245 (+942948us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:44.546316 (+    71us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:44.546390 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:44.546486 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:44.546626 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4646,"prepare.queue_time_us":191634,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"thread_start_us":1260083,"threads_started":2}
W20260709 13:02:44.544551  8584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.015s	sys 0.031s
I20260709 13:02:44.370834 11738 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.372385 11692 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.450130 11816 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:44.555279  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:44.555763  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:44.559391  8658 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:44.403656  8466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 218872us
I20260709 13:02:44.580824  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:44.497052 11812 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:44.560501  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:44.368721 11875 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:44.563206  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:44.355616 11876 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.379106 11731 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:44.373865 11822 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:44.377614 11617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.978s	user 0.030s	sys 0.037s
I20260709 13:02:44.486574 11704 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.584113  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:44.584406  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:44.585367  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:44.551959 11911 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:44.385699 11892 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.196s
I20260709 13:02:44.597608  8655 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:44.561069 11864 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:44.387279 11893 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.602506 11890 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:44.567781 11952 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:44.382468 11837 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.601310  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:44.631263  9328 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:44.396888 11951 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.569438 11816 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:44.386161 11579 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:44.609270  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:44.613739  8725 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:44.618353 11892 consensus_queue.cc:1243] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:44.615330  8653 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:44.593281 11604 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:44.386801 11980 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.619514 11865 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:44.552179 11783 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:44.607997 11954 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.992s	user 0.020s	sys 0.032s
I20260709 13:02:44.569110 11738 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:44.654059  9327 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:44.654579  8657 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:44.612448 11878 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:44.619614 11886 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.007s	sys 0.028s
W20260709 13:02:44.642400  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 159419us
W20260709 13:02:44.643718 11954 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.042s	user 0.022s	sys 0.033s
I20260709 13:02:44.647101 11692 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:44.643327 11960 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:44.647569 11837 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:44.668895  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:44.647856 11812 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:44.664707  8733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.003s	sys 0.007s
I20260709 13:02:44.649335 11876 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:44.649888 11937 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 2.190s	user 0.014s	sys 0.000s
I20260709 13:02:44.650534  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 614) took 2484 ms. Trace:
I20260709 13:02:44.640077 11867 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:44.662766 11886 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.008s	sys 0.029s
I20260709 13:02:44.625751 11864 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:44.606949 11731 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:44.626480  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:44.665445  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:44.687693  8658 raft_consensus.cc:2376] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8. Re-sending same reply.
W20260709 13:02:44.688434  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.688711  8731 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:44.689121  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.670409  8725 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:44.612762 11704 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:44.675310  8733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.003s	sys 0.007s
W20260709 13:02:44.651844 11986 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:02:44.679585  9331 rpcz_store.cc:276] 0709 13:02:42.165660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.267283 (+101623us) service_pool.cc:224] Handling call
0709 13:02:42.286915 (+ 19632us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:42.297813 (+ 10898us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:42.297859 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:43.864497 (+1566638us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b2c00047a38. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:02:43.865003 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 13:02:44.325560 (+460557us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:44.325596 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:44.325675 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:44.650198 (+324523us) raft_consensus.cc:1685] finished
0709 13:02:44.650226 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:44.650467 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10348,"raft.queue_time_us":15961,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5281,"spinlock_wait_cycles":31238912,"thread_start_us":460102,"threads_started":1,"wal-append.queue_time_us":742411}
I20260709 13:02:44.661270 11865 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:44.685936  8622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.072s	user 0.037s	sys 0.025s
I20260709 13:02:44.661558 11783 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:45.236097  9322 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.007s	sys 0.007s
W20260709 13:02:45.299594  8579 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.013s	sys 0.017s
W20260709 13:02:45.333709  8579 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.104s	user 0.013s	sys 0.017s
W20260709 13:02:45.110343  8656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.322s	user 0.019s	sys 0.013s
W20260709 13:02:45.109938  8729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.014s	sys 0.022s
I20260709 13:02:44.690577  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:44.736886  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:45.334921  8729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.989s	user 0.014s	sys 0.029s
I20260709 13:02:45.335160  9327 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:44.769088  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:44.995817 11785 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.577s	user 0.002s	sys 0.007s
W20260709 13:02:44.760879  8625 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.007s	sys 0.012s
W20260709 13:02:45.296766 11991 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 13:02:45.255993  8659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.010s	sys 0.001s
I20260709 13:02:44.686448 11886 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:44.693776 11979 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:44.684731 11878 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.679411 11972 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:44.992409 11731 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:45.337003  9322 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.008s	sys 0.008s
I20260709 13:02:45.307932  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:44.736382 11816 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:45.137092  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.008s	sys 0.027s
W20260709 13:02:45.353534  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.013s	sys 0.028s
W20260709 13:02:45.213488  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.830s	user 0.004s	sys 0.033s
W20260709 13:02:45.354210  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.004s	sys 0.036s
I20260709 13:02:44.683008 11812 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:45.311369 11992 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 13:02:45.311336 11807 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:45.264274  8730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.000s	sys 0.032s
W20260709 13:02:45.312068 11955 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.008s	sys 0.029s
W20260709 13:02:44.996352  8549 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.870s	user 0.002s	sys 0.032s
W20260709 13:02:45.376933  8549 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.250s	user 0.013s	sys 0.032s
W20260709 13:02:45.377350  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.36975e+06us
W20260709 13:02:45.379598  8549 connection.cc:256] RPC call timeout handler was delayed by -1.2738993159999836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
I20260709 13:02:44.756656 11604 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:44.887761 11996 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
W20260709 13:02:45.109964 11843 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.156s	user 0.011s	sys 0.011s
W20260709 13:02:45.195207  9330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.014s	sys 0.006s
I20260709 13:02:45.255829 11738 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:45.231438 11793 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.877s	user 0.012s	sys 0.000s
W20260709 13:02:44.909128  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.388778  8650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.008s	sys 0.027s
W20260709 13:02:45.091719  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.385845  9330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.016s	user 0.015s	sys 0.006s
I20260709 13:02:45.392699  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:45.393484  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.393947  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.330868 11865 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.394681  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:45.320431 11977 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.000s	sys 0.010s
W20260709 13:02:45.335583  8656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.547s	user 0.019s	sys 0.021s
W20260709 13:02:44.880674 11969 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.018s	sys 0.019s
W20260709 13:02:45.336330  8625 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.255s	user 0.010s	sys 0.031s
W20260709 13:02:45.399762  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 1.31868e+06us
W20260709 13:02:44.754597  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.401319  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.335804  8658 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 8 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 13:02:44.790760 11871 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.815s	user 0.024s	sys 0.027s
I20260709 13:02:45.336961  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1968) took 3642 ms. Trace:
W20260709 13:02:45.109673 11958 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.008s	sys 0.013s
W20260709 13:02:44.770706 11593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.010s	sys 0.025s
W20260709 13:02:45.339496  8659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.011s	sys 0.001s
I20260709 13:02:45.325744 11783 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:45.342103 11704 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:45.342664  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:44.890138 11579 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:45.345649 11972 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.340430 11785 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.348346 11909 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.370052  9327 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:45.336140 11864 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:45.354609  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 970889us
W20260709 13:02:45.374275  8730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.000s	sys 0.034s
I20260709 13:02:44.668973 11960 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:45.412755 11871 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.457s	user 0.029s	sys 0.041s
W20260709 13:02:45.376392 11955 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.989s	user 0.009s	sys 0.030s
W20260709 13:02:45.302923  8622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.689s	user 0.041s	sys 0.046s
W20260709 13:02:45.395937  8650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.008s	sys 0.027s
I20260709 13:02:45.413170  8727 rpcz_store.cc:276] 0709 13:02:41.694198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.696401 (+  2203us) service_pool.cc:224] Handling call
0709 13:02:41.749428 (+ 53027us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.749714 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.749754 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.794654 (+ 44900us) log.cc:844] Serialized 26 byte log entry
0709 13:02:42.344108 (+549454us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:42.344164 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.344339 (+   175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:44.641765 (+2297426us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:44.791326 (+149561us) raft_consensus.cc:1685] finished
0709 13:02:44.791353 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.336863 (+545510us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3283328,"thread_start_us":380857,"threads_started":1,"wal-append.queue_time_us":538116}
I20260709 13:02:45.391402 11786 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.419843  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2399) took 1955 ms. Trace:
I20260709 13:02:45.413352 11929 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.385782 11843 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.433s	user 0.011s	sys 0.014s
I20260709 13:02:45.432492  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:45.388734  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.098s	user 0.008s	sys 0.030s
W20260709 13:02:45.455747  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.165s	user 0.008s	sys 0.030s
W20260709 13:02:45.456116  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.16564e+06us
I20260709 13:02:45.388206 11816 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:45.387815 11849 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.414417 11969 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.325s	user 0.031s	sys 0.024s
I20260709 13:02:45.388389 11889 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.392558 12000 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 13:02:45.389694 12016 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.398555 11977 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.000s	sys 0.010s
I20260709 13:02:45.345294 11979 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.441089  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:45.423125 11958 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.046s	user 0.008s	sys 0.016s
W20260709 13:02:45.423369 11593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.222s	user 0.015s	sys 0.040s
I20260709 13:02:45.444058  8584 rpcz_store.cc:276] 0709 13:02:43.464537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.520343 (+ 55806us) service_pool.cc:224] Handling call
0709 13:02:43.520687 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:43.521122 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:43.521157 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:43.521686 (+   529us) log.cc:844] Serialized 26 byte log entry
0709 13:02:44.550057 (+1028371us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:44.550117 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:44.550211 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:45.419596 (+869385us) raft_consensus.cc:1685] finished
0709 13:02:45.419618 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.419774 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5004,"raft.queue_time_us":6844,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":56875,"thread_start_us":867007,"threads_started":1,"wal-append.queue_time_us":1300013}
I20260709 13:02:45.350996 11878 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.649566 11812 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:45.650874 11878 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:45.443243  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 1.82932e+06us
W20260709 13:02:45.651458 12003 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
W20260709 13:02:45.654345 12009 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
W20260709 13:02:45.443306  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 142598us
W20260709 13:02:45.461843  8698 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.523s (SENT)
W20260709 13:02:45.462455  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.656962  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:45.657912  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 196071us
W20260709 13:02:45.659116  8698 leader_election.cc:336] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.811s (SENT)
I20260709 13:02:45.660275  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:45.424943 11793 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:45.661213 11812 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.661624 11793 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.435639 11783 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.438591 11960 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:45.466709 11731 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 13:02:45.458726 11903 raft_consensus.cc:2764] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:45.465499 11786 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.442566 11864 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.443972 11909 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.494122 12010 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:45.430044 11886 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.467435 11849 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.465694 11929 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.485646 12004 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.488636 11824 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:45.695703  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:45.664462  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:45.664875  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:45.664023  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:45.662683  9323 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:02:45.706758  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:45.665446  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:45.748116  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2427) took 1128 ms. Trace:
I20260709 13:02:45.748968  8658 rpcz_store.cc:276] 0709 13:02:44.619775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.575151 (+955376us) service_pool.cc:224] Handling call
0709 13:02:45.748057 (+172906us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.739387  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:45.750097  8576 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 3.
I20260709 13:02:45.751366  8580 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:45.662766  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 821) took 4938 ms. Trace:
W20260709 13:02:45.662415 12019 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260709 13:02:45.754995  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2417) took 1161 ms. Trace:
I20260709 13:02:45.755529  8650 rpcz_store.cc:276] 0709 13:02:44.593522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.596626 (+  3104us) service_pool.cc:224] Handling call
0709 13:02:44.614085 (+ 17459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:44.614734 (+   649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:44.614776 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:44.626374 (+ 11598us) log.cc:844] Serialized 26 byte log entry
0709 13:02:45.443681 (+817307us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:45.443719 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.443783 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:45.754730 (+310947us) raft_consensus.cc:1685] finished
0709 13:02:45.754758 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.754922 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13450,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":5108,"thread_start_us":787524,"threads_started":1,"wal-append.queue_time_us":872948}
I20260709 13:02:45.752923  9325 rpcz_store.cc:276] 0709 13:02:40.724311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.935772 (+211461us) service_pool.cc:224] Handling call
0709 13:02:42.777975 (+1842203us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:42.781196 (+  3221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.086101 (+1304905us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:44.086146 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:44.086203 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:44.086254 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.662508 (+1576254us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400693054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:45.662553 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.662682 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3004809856}
I20260709 13:02:45.699178 11878 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:45.684939 11958 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:45.662164 11793 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:45.768814  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62157.1us
I20260709 13:02:45.767897  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:45.782267  8732 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:45.783286  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 638) took 1163 ms. Trace:
I20260709 13:02:45.783753  8732 rpcz_store.cc:276] 0709 13:02:44.619428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.748785 (+1129357us) service_pool.cc:224] Handling call
0709 13:02:45.782017 (+ 33232us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.782907 (+   890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.783203 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.754035  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:45.661931 11883 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:45.650923 11785 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:45.661784 11812 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.779171  8625 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:45.717568 11807 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.803229  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:45.785147  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 390456us
W20260709 13:02:45.794703  8549 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.605s (SENT)
I20260709 13:02:45.808873  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:45.810860  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.797487  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43537.9us
I20260709 13:02:45.641968 11889 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.819451  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2309) took 2351 ms. Trace:
I20260709 13:02:45.819901  8733 rpcz_store.cc:276] 0709 13:02:43.468132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.822556 (+354424us) service_pool.cc:224] Handling call
0709 13:02:43.866438 (+ 43882us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:43.866972 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:43.867026 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:43.901306 (+ 34280us) log.cc:844] Serialized 52 byte log entry
0709 13:02:45.298976 (+1397670us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:45.751708 (+452732us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.779083 (+ 27375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.779162 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:45.819238 (+ 40076us) raft_consensus.cc:1685] finished
0709 13:02:45.819256 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.819392 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27137,"raft.queue_time_us":60769,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":25666,"thread_start_us":1774908,"threads_started":2,"wal-append.queue_time_us":1328589}
I20260709 13:02:45.753322 11938 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.776005 11972 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:45.575456 12016 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:45.815259 12014 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 13:02:45.839421  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:45.777997 11704 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:45.840538  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 447839us
I20260709 13:02:45.752450 11824 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.713349 11783 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.846237  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2295) took 2580 ms. Trace:
I20260709 13:02:45.764730 11864 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.849113  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:45.825531 11958 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:45.851598  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2432) took 1181 ms. Trace:
I20260709 13:02:45.847756 11704 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.822337 11955 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:45.826264  8696 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:45.864809  8658 rpcz_store.cc:276] 0709 13:02:44.670625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.794065 (+1123440us) service_pool.cc:224] Handling call
0709 13:02:45.851522 (+ 57457us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.748922 12004 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.847505 11843 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:45.839514 12020 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:45.846679 11865 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:45.864244 11958 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:45.871165 12030 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:45.849956 11972 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.872988  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46845us
I20260709 13:02:45.825682  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 634) took 2342 ms. Trace:
I20260709 13:02:45.874106 11704 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:45.878253 11873 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:45.874894  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:45.840755 11593 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 13:02:45.825161 11807 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:45.870064  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.907025  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:45.883762  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:45.914966  8658 raft_consensus.cc:2376] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6. Re-sending same reply.
I20260709 13:02:45.915460  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2421) took 1319 ms. Trace:
I20260709 13:02:45.915760  8658 rpcz_store.cc:276] 0709 13:02:44.595855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.869185 (+1273330us) service_pool.cc:224] Handling call
0709 13:02:45.915406 (+ 46221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.841388 11938 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.918056  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:45.897444  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.898017  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 94841.5us
I20260709 13:02:45.848685 11579 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:45.866529 12035 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:45.916797  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:45.923151  9331 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:45.918709  9327 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:45.919581  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:45.925187  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:45.879786 12029 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:45.928581  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:45.904886 11958 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:45.873529 11731 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.931341  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:45.920120 12011 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.421s	user 0.000s	sys 0.001s
I20260709 13:02:45.909000 11843 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:45.903632  8729 rpcz_store.cc:276] 0709 13:02:43.483682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.893841 (+410159us) service_pool.cc:224] Handling call
0709 13:02:44.175190 (+281349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.175473 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:44.345676 (+170203us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.825242 (+1479566us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.825307 (+    65us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:45.825359 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.825450 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.825598 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":133397,"prepare.queue_time_us":204247,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":94254,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":22072,"thread_start_us":1479132,"threads_started":2,"wal-append.queue_time_us":1122192}
I20260709 13:02:45.913239 11955 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:45.927192 12008 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:45.935976  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:45.944867  8727 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:45.912518 11900 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.950001 11994 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.630s	user 0.005s	sys 0.001s
I20260709 13:02:45.950218 11900 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.933584  9325 raft_consensus.cc:2410] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 13:02:45.939774  8697 leader_election.cc:336] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.811s (SENT)
I20260709 13:02:45.946223 11578 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:45.947270 11875 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:45.947342 12017 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 13:02:45.926527 12035 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.951030  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2304) took 2129 ms. Trace:
I20260709 13:02:45.974730  8651 rpcz_store.cc:276] 0709 13:02:43.823549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.827569 (+  4020us) service_pool.cc:224] Handling call
0709 13:02:45.950316 (+2122747us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:02:45.950515 (+   199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.950678 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.950711 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.950749 (+    38us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:45.950781 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.950855 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.950969 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75616640}
W20260709 13:02:45.937003 11899 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.602s	user 0.010s	sys 0.005s
I20260709 13:02:45.958588  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:45.961910  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:45.898190 12020 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:45.964937  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.912372 11972 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.953516 12008 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:45.984822 12022 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 13:02:45.929258 11579 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:45.931747 11873 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:45.944243 12046 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:45.991027  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2310) took 2164 ms (client timeout 1822 ms). Trace:
I20260709 13:02:46.606611  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2287) took 4834 ms. Trace:
W20260709 13:02:46.537951  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.332716  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:46.618167  8654 rpcz_store.cc:269] 0709 13:02:43.828270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.120759 (+292489us) service_pool.cc:224] Handling call
0709 13:02:45.990822 (+1870063us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400131ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:45.990950 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414200576}
W20260709 13:02:46.606835  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 667151us
I20260709 13:02:46.412904  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 830) took 2073 ms. Trace:
W20260709 13:02:46.522207  9297 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.007s	sys 0.003s
W20260709 13:02:46.604547  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 625) took 2020 ms (client timeout 1811 ms). Trace:
I20260709 13:02:46.006552  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2418) took 1412 ms. Trace:
W20260709 13:02:46.607745  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 812966us
I20260709 13:02:46.602902  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:46.621156  9297 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.011s	sys 0.003s
I20260709 13:02:46.621737  8650 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:46.621886  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 832) took 2281 ms. Trace:
W20260709 13:02:46.622463  8650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2431) took 1978 ms (client timeout 1962 ms). Trace:
W20260709 13:02:46.623060  8650 rpcz_store.cc:269] 0709 13:02:44.644163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.602528 (+1958365us) service_pool.cc:224] Handling call
0709 13:02:46.622396 (+ 19868us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.596948  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1841) took 3568 ms. Trace:
I20260709 13:02:46.560539  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 482) took 2695 ms. Trace:
I20260709 13:02:46.352617  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:46.249770  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:46.624318  8579 rpcz_store.cc:276] 0709 13:02:43.028761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.447176 (+418415us) service_pool.cc:224] Handling call
0709 13:02:44.212776 (+765600us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.213125 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:45.334352 (+1121227us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.769441 (+435089us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.769495 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:45.769541 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.769611 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.596680 (+827069us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54000f87d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:46.596852 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14709,"prepare.queue_time_us":394934,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":45398,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9520,"spinlock_wait_cycles":1759763328,"thread_start_us":1524445,"threads_started":2,"wal-append.queue_time_us":664828}
I20260709 13:02:46.624447  8652 rpcz_store.cc:276] 0709 13:02:43.864713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.447773 (+583060us) service_pool.cc:224] Handling call
0709 13:02:44.448200 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.669630 (+221430us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:44.669673 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:44.669731 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:44.669782 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.560171 (+1890389us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b540018fbd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:46.560248 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.560454 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2407,"raft.queue_time_us":2568,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":12212,"spinlock_wait_cycles":502462848}
I20260709 13:02:46.607394  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:46.628113  9325 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:46.178335  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.349050 12020 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:46.249301  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:46.620450  9323 rpcz_store.cc:269] 0709 13:02:44.584038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.584163 (+   125us) service_pool.cc:224] Handling call
0709 13:02:46.604322 (+2020159us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54007dc3d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:46.604478 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2427743104}
W20260709 13:02:46.333562  8578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 627) took 2510 ms (client timeout 2040 ms). Trace:
W20260709 13:02:46.630241  8578 rpcz_store.cc:269] 0709 13:02:43.822842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.822964 (+   122us) service_pool.cc:224] Handling call
0709 13:02:46.333253 (+2510289us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b5400179b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:46.333468 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66553088}
I20260709 13:02:46.328977  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 485) took 2434 ms. Trace:
I20260709 13:02:46.622270  9330 rpcz_store.cc:276] 0709 13:02:44.340331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.341201 (+   870us) service_pool.cc:224] Handling call
0709 13:02:44.341645 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:44.365118 (+ 23473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:44.365148 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:44.369190 (+  4042us) log.cc:844] Serialized 26 byte log entry
0709 13:02:45.392589 (+1023399us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:45.392634 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.392704 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:46.621662 (+1228958us) raft_consensus.cc:1685] finished
0709 13:02:46.621682 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.621827 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22976,"raft.queue_time_us":36826,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":1022968,"threads_started":1,"wal-append.queue_time_us":1089068}
W20260709 13:02:46.623800  8549 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.705s (SENT)
I20260709 13:02:46.631843  8653 rpcz_store.cc:276] 0709 13:02:43.894838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.470042 (+575204us) service_pool.cc:224] Handling call
0709 13:02:44.549222 (+ 79180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.637491 (+ 88269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.328562 (+1691071us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b540026b954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:46.328869 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151473920}
I20260709 13:02:46.618925  9326 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:46.634037  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:46.619503  9329 rpcz_store.cc:276] 0709 13:02:44.339578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.339752 (+   174us) service_pool.cc:224] Handling call
0709 13:02:44.340127 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.423566 (+ 83439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:44.423598 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:44.423642 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:44.423678 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.412467 (+1988789us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b5400541d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:46.412583 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.412807 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":390,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":191099648}
W20260709 13:02:46.621047  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.958186 11786 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:46.622869  9297 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:45.963112 12024 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.989253 11816 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.527s	user 0.005s	sys 0.002s
W20260709 13:02:46.712744  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2423) took 2116 ms (client timeout 1910 ms). Trace:
I20260709 13:02:45.963256 11955 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:46.716231  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:46.621112  8659 rpcz_store.cc:276] 0709 13:02:44.593902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.594043 (+   141us) service_pool.cc:224] Handling call
0709 13:02:44.594508 (+   465us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:44.604442 (+  9934us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:44.604489 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:44.605335 (+   846us) log.cc:844] Serialized 52 byte log entry
0709 13:02:45.419282 (+813947us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:45.848468 (+429186us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.945983 (+ 97515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.957030 (+ 11047us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:46.006227 (+ 49197us) raft_consensus.cc:1685] finished
0709 13:02:46.006257 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.006471 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35986,"prepare.queue_time_us":499857,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":36168,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":8471,"thread_start_us":1282183,"threads_started":3,"wal-append.queue_time_us":1048137}
I20260709 13:02:46.713156  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:46.705964  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:46.332520 12045 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:45.957479 11731 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:46.330384 11807 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:45.989717 11812 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:46.901103  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.975s	user 0.014s	sys 0.006s
I20260709 13:02:46.616003  8656 rpcz_store.cc:276] 0709 13:02:41.744772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.745637 (+   865us) service_pool.cc:224] Handling call
0709 13:02:41.788710 (+ 43073us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:43.787924 (+1999214us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:43.788252 (+   328us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.855805 (+2067553us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.875639 (+ 19834us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.875710 (+    71us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:45.875782 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.354727 (+478945us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b180042acc0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f
0709 13:02:46.356324 (+  1597us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b48001b2358. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:02:46.578536 (+222212us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b540013a454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:46.578600 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.578798 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22742,"prepare.queue_time_us":500021,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":2980,"raft.queue_time_us":113470,"raft.run_cpu_time_us":3590,"raft.run_wall_time_us":457724,"spinlock_wait_cycles":4612110336,"thread_start_us":2063627,"threads_started":2,"wal-append.queue_time_us":1894877}
I20260709 13:02:46.777515  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:46.892164  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:46.833976 11842 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.009s	sys 0.009s
I20260709 13:02:46.676147  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:46.718288  8651 rpcz_store.cc:269] 0709 13:02:44.596451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.976425 (+1379974us) service_pool.cc:224] Handling call
0709 13:02:46.712662 (+736237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.985183 11864 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:46.210141 11692 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:46.637629 12008 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:46.959053  8652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2419) took 2364 ms (client timeout 2612 ms). Trace:
W20260709 13:02:46.979223  8652 rpcz_store.cc:269] 0709 13:02:44.595220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.668796 (+2073576us) service_pool.cc:224] Handling call
0709 13:02:46.958984 (+290188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.980175  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:46.960479  8582 raft_consensus.cc:2410] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 13:02:46.960567  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.016s	sys 0.007s
I20260709 13:02:45.979528 12035 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:46.962182  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:46.350692 11892 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.981384  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:46.983142  8578 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:46.668856 12030 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.847833  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:46.610723 11960 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.860s	user 0.009s	sys 0.003s
I20260709 13:02:46.645387  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:46.118659 11793 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:46.669306  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46516.4us
I20260709 13:02:46.762030 12036 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:46.861101  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:46.644011 11604 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.002s	sys 0.006s
I20260709 13:02:46.558920 11951 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:46.572602 11958 consensus_meta.cc:287] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.632s	user 0.004s	sys 0.006s
I20260709 13:02:45.991182 12038 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:46.642863  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:46.669342  9326 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 626) took 2050 ms (client timeout 1837 ms). Trace:
I20260709 13:02:46.959784  8654 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:46.638635  8549 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.815s (SENT)
I20260709 13:02:46.329452 12004 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:46.537573 11883 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.011s	sys 0.017s
I20260709 13:02:46.021819 11875 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:46.761698  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 89908.6us
I20260709 13:02:46.214489 11558 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:46.085204 11878 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:45.992908 11660 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:46.701278 11679 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:46.006738 12029 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:46.637313 11783 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:46.330104 11891 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:45.992069 12041 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:02:46.395009 11980 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:46.769376 11569 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.222082 11909 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:45.999179 11979 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:46.558799 12016 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:46.713006 11903 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:46.716506 11911 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.742017 12046 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:46.192044 11865 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:46.714250 11798 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:46.717250 11797 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:47.192884 11889 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:46.998472 11892 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:47.167409  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 397756us
I20260709 13:02:47.197934 11892 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:46.006047 11899 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.199235 11731 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.013161  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:46.975323  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.201848 11793 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.202427 11899 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.202829 11878 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:47.203207 11793 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:47.203697 11878 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:47.204129  9324 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:47.203847  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:46.981923  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 1.05666e+06us
I20260709 13:02:47.008172  8578 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 13:02:47.207146  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 627) took 2622 ms (client timeout 2862 ms). Trace:
W20260709 13:02:47.207722  9324 rpcz_store.cc:269] 0709 13:02:44.584918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.585075 (+   157us) service_pool.cc:224] Handling call
0709 13:02:47.206802 (+2621727us) spinlock_profiling.cc:243] Waited 2.59 s on lock 0x7b54005dacd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:47.207004 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5957995904}
I20260709 13:02:46.839128  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.001076  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.210398 11886 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:47.212603  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:46.987100  8657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2309) took 3151 ms (client timeout 2008 ms). Trace:
I20260709 13:02:47.005134  8581 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:47.214083 11886 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:47.213855  8657 rpcz_store.cc:269] 0709 13:02:43.826774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.247302 (+420528us) service_pool.cc:224] Handling call
0709 13:02:46.978325 (+2731023us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b540007dad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:46.978445 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":815871744}
I20260709 13:02:47.214728 11886 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:47.215593  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.216601  8657 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:46.970515  8650 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:47.137354 11929 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:47.217976  8650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2429) took 2589 ms (client timeout 2203 ms). Trace:
I20260709 13:02:47.049307 11955 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.012491  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:47.218515  8650 rpcz_store.cc:269] 0709 13:02:44.636176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.667831 (+2031655us) service_pool.cc:224] Handling call
0709 13:02:47.217676 (+549845us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:47.219815 11849 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.220067  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:47.220314  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:45.854185  8730 rpcz_store.cc:276] 0709 13:02:43.265892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.279608 (+ 13716us) service_pool.cc:224] Handling call
0709 13:02:43.279904 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:43.280109 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:43.511655 (+231546us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.841805 (+2330150us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.845689 (+  3884us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.845766 (+    77us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:45.845856 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.845947 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.846128 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3441,"prepare.queue_time_us":192037,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":26,"raft.queue_time_us":805359,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":6560,"spinlock_wait_cycles":51788160,"thread_start_us":2554114,"threads_started":4,"wal-append.queue_time_us":1484766}
I20260709 13:02:46.970485  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:46.961959  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.222651  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.223362  8659 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 8 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:47.223248  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:47.031098  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2425) took 2427 ms (client timeout 2144 ms). Trace:
I20260709 13:02:47.224207  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.094342  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:46.989081  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2430) took 2349 ms (client timeout 2862 ms). Trace:
I20260709 13:02:47.226076  9325 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 5: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:47.224236  8656 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
W20260709 13:02:47.226284  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 213094us
I20260709 13:02:47.228189  8656 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:47.228677  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:47.224618  8654 rpcz_store.cc:269] 0709 13:02:44.603993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.705611 (+2101618us) service_pool.cc:224] Handling call
0709 13:02:47.030982 (+325371us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:47.232096  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 834) took 2890 ms. Trace:
I20260709 13:02:47.116847 11903 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:47.037068  9296 leader_election.cc:336] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.866s (SENT)
W20260709 13:02:46.906744  8733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.976s	user 0.012s	sys 0.075s
I20260709 13:02:47.179840  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 481) took 3315 ms. Trace:
W20260709 13:02:47.024713  9326 rpcz_store.cc:269] 0709 13:02:44.584398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.332338 (+1747940us) service_pool.cc:224] Handling call
0709 13:02:46.634734 (+302396us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.995396 11960 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.002995  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:47.099246 11660 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:47.236028 11903 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:47.232671  9322 rpcz_store.cc:276] 0709 13:02:44.341059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.354326 (+ 13267us) service_pool.cc:224] Handling call
0709 13:02:44.374508 (+ 20182us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.449281 (+ 74773us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:45.351417 (+902136us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.714832 (+363415us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.714901 (+    69us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:45.714969 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.133628 (+1418659us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:47.193043 (+ 59415us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x7b54003b01d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:47.193182 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.231813 (+ 38631us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1260822,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":61611,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":212,"spinlock_wait_cycles":194450304,"thread_start_us":1161627,"threads_started":2,"wal-append.queue_time_us":557412}
I20260709 13:02:47.125658 11865 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:47.237360  8733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.306s	user 0.016s	sys 0.078s
I20260709 13:02:47.236938 11660 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.238772 11660 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:47.239173 11865 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:47.238660  8655 rpcz_store.cc:276] 0709 13:02:43.864413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.390629 (+526216us) service_pool.cc:224] Handling call
0709 13:02:44.390925 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.617030 (+226105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.179040 (+2562010us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540018f6d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:47.179527 (+   487us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462107904}
W20260709 13:02:47.225849  8653 rpcz_store.cc:269] 0709 13:02:44.639881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.669387 (+2029506us) service_pool.cc:224] Handling call
0709 13:02:46.989019 (+319632us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:47.122318 11911 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.241662  8653 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 13:02:45.988018 11843 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:47.243878  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 240808us
I20260709 13:02:47.244665 11911 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:46.971130 11842 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s	user 0.009s	sys 0.016s
I20260709 13:02:47.245791 11806 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:47.175382  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1837) took 4201 ms. Trace:
I20260709 13:02:46.314945 11704 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.246654  8583 rpcz_store.cc:276] 0709 13:02:42.973530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.973785 (+   255us) service_pool.cc:224] Handling call
0709 13:02:42.995803 (+ 22018us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:43.034601 (+ 38798us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:43.034632 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:43.035216 (+   584us) log.cc:844] Serialized 52 byte log entry
0709 13:02:43.296183 (+260967us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:44.292233 (+996050us) log.cc:844] Serialized 14 byte log entry
0709 13:02:44.292562 (+   329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.173843 (+2881281us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b5400168ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:47.173995 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.174153 (+   158us) raft_consensus.cc:1685] finished
0709 13:02:47.174526 (+   373us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.175076 (+   550us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1268367,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":5958,"raft.queue_time_us":38125,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":46156,"spinlock_wait_cycles":88313216,"thread_start_us":1188940,"threads_started":2,"wal-append.queue_time_us":564076,"wal-append.run_cpu_time_us":3136,"wal-append.run_wall_time_us":1813436}
I20260709 13:02:47.248920 11960 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.205248  8577 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:47.190249  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:45.987624 11824 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:46.983500 11692 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:47.252187  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62321.3us
I20260709 13:02:47.221346  8650 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:47.253633 11806 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:47.254808 11692 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.133097 11938 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.255277 11889 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:47.224604  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.124739 11875 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.256325 11938 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:47.257282  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.257730 11938 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:47.258167  8659 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:47.258272 11938 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:44.327185  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:47.259944 11938 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.218289 11900 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.210428  8580 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:47.030400 11958 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.112860 11569 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:47.187428  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.262964  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.096015 11558 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:47.249955  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:47.265048  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 228024us
I20260709 13:02:47.031641  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:47.266017 11731 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:47.178939  8479 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 4 to 5, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 5 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:47.267410  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 628567us
I20260709 13:02:47.249959 11960 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:46.959394 11893 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.228901  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2424) took 2630 ms. Trace:
I20260709 13:02:47.254710  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.264989 11569 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:47.172075  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.118161 11972 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.253785 11865 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:47.269130  8656 rpcz_store.cc:276] 0709 13:02:44.598213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.227090 (+2628877us) service_pool.cc:224] Handling call
0709 13:02:47.227844 (+   754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.228651 (+   807us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.228816 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:47.264946 11958 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:47.254979  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:47.270999 11900 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:47.271600 11849 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:47.271885 11938 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:47.272248 11579 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.233982  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:47.130609 11988 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.912s	user 0.003s	sys 0.000s
W20260709 13:02:46.422883 11998 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.602s	user 0.003s	sys 0.010s
W20260709 13:02:46.996174 12012 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.427s	user 0.000s	sys 0.003s
I20260709 13:02:47.250406 11704 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:46.932427 12020 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:47.134168 12028 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.213s	user 0.000s	sys 0.003s
I20260709 13:02:47.010901 12030 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.385006 12020 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.386593 12008 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.387840 12020 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.388628 12030 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:46.772271 12024 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.037945 12038 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.425585  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.426429  9324 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:47.428395 12038 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.428664 11930 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.429618  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.430433  9331 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:47.430799  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.431488  9330 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:47.431903  9331 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:47.119602 12052 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260709 13:02:47.130993 12053 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
W20260709 13:02:47.113495 12056 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 13:02:47.435891  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2320) took 1527 ms. Trace:
I20260709 13:02:47.436323  8733 rpcz_store.cc:276] 0709 13:02:45.908755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.910185 (+  1430us) service_pool.cc:224] Handling call
0709 13:02:45.926339 (+ 16154us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:45.926552 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.926580 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:45.927087 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 13:02:47.238131 (+1311044us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:47.238384 (+   253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.238518 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.435518 (+197000us) raft_consensus.cc:1685] finished
0709 13:02:47.435542 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.435741 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1306808,"threads_started":1,"wal-append.queue_time_us":1052036}
I20260709 13:02:47.123615 12046 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:47.011873 11604 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.916s	user 0.003s	sys 0.019s
I20260709 13:02:47.438153 11903 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.260504  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.216238 11886 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.440577 11903 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.441243 11886 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:47.442313  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:46.966007 12045 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.269737  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:47.261125 11873 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:47.331456  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 76676.6us
W20260709 13:02:47.444936  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 191852us
I20260709 13:02:47.446446  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:47.447083  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:47.332376 11960 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:47.327253 11849 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:47.448593 12030 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:47.449345 11849 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.450651 12030 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:47.451131 11849 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:47.344584  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.453881  9326 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:47.453979  8653 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:47.359918  8581 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:47.353540  8549 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.879s (SENT)
I20260709 13:02:47.455840  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:47.365615  8650 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:47.265332 11558 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.457623 11558 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:47.457827  8698 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.605s (SENT)
I20260709 13:02:47.458516  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:47.459343 12024 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.460098 12024 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:47.381946  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.014242 12036 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.462410 12046 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.462599 11960 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:47.388893 12008 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:47.463093 11960 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.372927 11865 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:47.463521 12036 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.464540 11865 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.418037  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.422714  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.421998  8479 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 5 to 6, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 6 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:47.423868  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.424429  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.467955 12057 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:47.429384 11930 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.139230 12035 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.447995 11955 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:47.482725 12057 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.484639  9327 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:47.484863 12035 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.485837  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.475596  9330 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:47.477723  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.419242  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.467965 11875 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.483214 11955 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:47.336407 11569 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.463551 11960 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:47.488018  9324 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:47.485600  9325 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:47.498493 11569 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.501775 11875 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.503365  8584 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:47.503635 11569 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.493623 11955 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:47.504069 11875 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.504674 12045 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.521818  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.522923  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.523885  8727 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:47.521827  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:47.506369  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:47.507418 12036 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:47.508495  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 156885us
I20260709 13:02:47.508924  8479 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 5, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 5 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:47.514988  8658 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:47.522167  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.529336  8724 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 13:02:47.531858  8549 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.972s (SENT)
I20260709 13:02:47.533279  8729 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:47.539676  9328 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:47.485237 12057 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:47.533751  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45154.4us
I20260709 13:02:47.533042  8479 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 3, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:47.543915  8480 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 302) took 30827 ms (client timeout 15000 ms). Trace:
I20260709 13:02:47.507088  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:47.546604  8575 raft_consensus.cc:2410] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:47.547243 11967 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.547544 11797 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:47.547997 11797 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.547778 11967 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.548497 11797 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.553612 11955 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.554960  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.556423 11960 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.556818  9327 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:47.557047  8653 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:47.558914  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:47.559378  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.537005  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:47.563735 11878 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.567365  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:47.569372  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2426) took 2956 ms (client timeout 1837 ms). Trace:
I20260709 13:02:47.543031  8479 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 reported cstate change: term changed from 4 to 5, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 5 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:47.579963  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.547699  9322 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:47.579998  9325 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:47.579277  8658 rpcz_store.cc:269] 0709 13:02:44.613259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.916365 (+1303106us) service_pool.cc:224] Handling call
0709 13:02:47.569069 (+1652704us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540016e4d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:47.569285 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3579642240}
I20260709 13:02:47.583788  8652 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 4.
I20260709 13:02:47.584065  8733 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:47.584358 11967 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:47.586565  9328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 630) took 2794 ms (client timeout 3512 ms). Trace:
W20260709 13:02:47.587162  9328 rpcz_store.cc:269] 0709 13:02:44.792662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.342302 (+549640us) service_pool.cc:224] Handling call
0709 13:02:47.586213 (+2243911us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b5400693a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:47.586492 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4863693696}
I20260709 13:02:47.587936 11960 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:47.590590  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48794.9us
I20260709 13:02:47.583987  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.593585  8651 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:47.595995 11930 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.596175 11955 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:47.597604  8659 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:47.606091  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:47.606371  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2405) took 1946 ms (client timeout 1907 ms). Trace:
W20260709 13:02:47.606738  8575 rpcz_store.cc:269] 0709 13:02:45.659419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.664617 (+  5198us) service_pool.cc:224] Handling call
0709 13:02:47.606061 (+1941444us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54000b5754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:47.606270 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4135446912}
I20260709 13:02:47.609782 11930 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:47.615136 11891 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 13:02:47.616493 11891 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:47.616946 11891 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:47.617746  8479 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 5, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 5 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:02:47.620375  8479 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 4, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:47.622232  8479 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:47.623687 11891 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:47.626394  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 94614.2us
I20260709 13:02:47.627341 11891 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:47.627558  8549 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.008s (SENT)
I20260709 13:02:47.628266  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:47.629562 11889 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:47.626441  8479 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 5, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 5 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:02:47.643921  8479 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:47.662879  8724 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:47.646581 11797 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.653496  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:47.663936  8729 raft_consensus.cc:2368] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5.
I20260709 13:02:47.654189  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.665040  9322 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:47.656278  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.657510  9330 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:47.657862  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.658670  9329 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:47.666639  8729 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:47.667845  9328 raft_consensus.cc:2410] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 13:02:47.659927  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52302us
I20260709 13:02:47.655649  9324 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:47.673276 11797 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.662680  8479 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 reported cstate change: term changed from 1 to 2, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:02:47.681964  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.692600  8733 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:47.786772  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:47.750694  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.824309  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2433) took 1224 ms. Trace:
I20260709 13:02:47.748119  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:47.693878  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:47.709911  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:47.730146  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:47.732990  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.695451  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:47.696875  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69387.3us
I20260709 13:02:47.836406  8724 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:47.787045  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:47.735198  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.728454  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.836869  9324 raft_consensus.cc:2376] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7. Re-sending same reply.
I20260709 13:02:47.842700  8658 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:47.696332  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.843185  8733 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:47.843837  8727 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:47.727463  8732 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:47.732859  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.840605  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.726540  8730 pending_rounds.cc:85] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:47.732170  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.856717  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.860273  8652 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:47.863579  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:47.864427  8659 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:47.867472  8728 raft_consensus.cc:2410] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:47.867558  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.867229  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.868418  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.868834  9329 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:47.868875  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.695766 11865 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:47.717331 11878 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:47.858274  8730 pending_rounds.cc:107] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 13:02:47.870376  8652 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:47.896212  8727 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:47.902110  8726 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:47.544593  8480 rpcz_store.cc:269] 0709 13:02:16.716711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.723465 (+  6754us) service_pool.cc:224] Handling call
0709 13:02:19.025999 (+2302534us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b540012a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b80090 00007f1409c1ef3a 00007f140f9d51cc 00007f140fad337e 00007f140df942a4 00007f140df94232 00007f140df94161 00007f140df940dc 00007f140df93392 00007f1407f5148c 00007f1407f508c6
0709 13:02:47.543389 (+28517390us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:33.947959 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:33.948579 (+   620us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:33.948641 (+    62us) write_op.cc:435] Acquired schema lock
0709 13:02:33.948654 (+    13us) tablet.cc:662] Decoding operations
0709 13:02:34.140074 (+191420us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:34.156144 (+ 16070us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:34.202946 (+ 46802us) tablet.cc:685] Acquiring locks for 6 operations
0709 13:02:34.375822 (+172876us) tablet.cc:701] Row locks acquired
0709 13:02:34.375847 (+    25us) write_op.cc:260] PREPARE: finished
0709 13:02:34.382021 (+  6174us) write_op.cc:270] Start()
0709 13:02:34.382192 (+   171us) write_op.cc:276] Timestamp: P: 1783602154375991 usec, L: 0
0709 13:02:34.382225 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 13:02:34.487234 (+105009us) log.cc:844] Serialized 2441 byte log entry
0709 13:02:35.796447 (+1309213us) op_driver.cc:464] REPLICATION: finished
0709 13:02:35.997052 (+200605us) write_op.cc:301] APPLY: starting
0709 13:02:35.997165 (+   113us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:35.997352 (+   187us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:36.024910 (+ 27558us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:36.077679 (+ 52769us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:36.080956 (+  3277us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0709 13:02:36.081007 (+    51us) write_op.cc:312] APPLY: finished
0709 13:02:36.087059 (+  6052us) log.cc:844] Serialized 53 byte log entry
0709 13:02:36.154092 (+ 67033us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:36.154219 (+   127us) write_op.cc:454] Released schema lock
0709 13:02:36.154275 (+    56us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:36.154735 (+   460us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:36.154763 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":745975,"reported_tablets":22,"spinlock_wait_cycles":393163648,"child_traces":[["op",{"apply.queue_time_us":170142,"apply.run_cpu_time_us":4009,"apply.run_wall_time_us":185126,"num_ops":6,"prepare.queue_time_us":94332,"prepare.run_cpu_time_us":14389,"prepare.run_wall_time_us":647893,"raft.queue_time_us":511720,"raft.run_cpu_time_us":42627,"raft.run_wall_time_us":119986,"replication_time_us":1401107,"thread_start_us":514028,"threads_started":4,"wal-append.queue_time_us":289373,"wal-append.run_cpu_time_us":194495,"wal-append.run_wall_time_us":2376463}]]}
I20260709 13:02:47.870704  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:47.884909  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.871138  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.912528  9328 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:47.835582  8654 rpcz_store.cc:276] 0709 13:02:46.599987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.230773 (+630786us) service_pool.cc:224] Handling call
0709 13:02:47.231424 (+   651us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:47.232296 (+   872us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:47.371385 (+139089us) log.cc:844] Serialized 14 byte log entry
0709 13:02:47.619592 (+248207us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:47.620269 (+   677us) log.cc:844] Serialized 26 byte log entry
0709 13:02:47.620431 (+   162us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:47.620691 (+   260us) log.cc:844] Serialized 14 byte log entry
0709 13:02:47.620920 (+   229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.621000 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.823845 (+202845us) raft_consensus.cc:1685] finished
0709 13:02:47.823880 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.824123 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37250,"prepare.queue_time_us":98396,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":116,"raft.queue_time_us":42835,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":362,"thread_start_us":310544,"threads_started":2,"wal-append.queue_time_us":295955}
I20260709 13:02:47.916095  8730 raft_consensus.cc:2889] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:47.916317  8575 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:47.907229  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:47.918401  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.919546  8728 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:47.920635  8725 raft_consensus.cc:2410] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:02:47.927201  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.928468  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.929195  8584 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:47.931109  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.936237 12008 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.937781  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:47.938165  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.939584 11604 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:47.941428  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:47.941782  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.942884  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:47.943089  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.943320  8583 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:47.943658  8580 raft_consensus.cc:2368] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7.
I20260709 13:02:47.929616 11865 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:47.951756  8658 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:47.953157  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.955861  8728 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:47.957885  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.958802  9329 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:47.959581  9328 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:47.961966  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.937042 12008 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:47.965425  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.969967  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.975261  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:47.966046  8732 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:47.985198  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.963754 11886 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.987816  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:47.993225  9324 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:47.990321  8728 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:47.993371  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.997800  9329 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:47.998863  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:47.974849 11891 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:48.003976  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.004695  9328 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:48.005527  8575 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:48.008020  8583 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:02:48.008451  8583 pending_rounds.cc:85] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 3
I20260709 13:02:48.008699  8583 pending_rounds.cc:107] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:02:48.008947  8583 raft_consensus.cc:2889] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:48.015377  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.024255  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.025835  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.008869 11900 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:48.034788 11900 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.036223  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:48.036842  8583 raft_consensus.cc:2376] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5. Re-sending same reply.
I20260709 13:02:48.037004 11900 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:48.037955  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.038300 11865 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:48.038714  8576 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:48.040944 12013 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:48.041913  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.042457  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:48.043241  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.044014  8731 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:48.044083  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:48.044871  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.045806  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.047622  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.048508  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.049489  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.046888  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.030864  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.046258  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.051779  8656 pending_rounds.cc:85] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 4
I20260709 13:02:48.052083  8656 pending_rounds.cc:107] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 13:02:48.052321  8656 raft_consensus.cc:2889] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:48.053126  8727 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:48.056620  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.059852  8658 raft_consensus.cc:2410] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:48.061452  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:48.064177  8655 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:48.065898  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.019618 11955 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:48.067113  8659 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:48.066004  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:48.071236  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.072870  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.073921  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.010133 11891 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.075619  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.105882 11660 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:02:48.106617  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:48.107177  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:48.108848  8650 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:48.114106  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.114090  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.114539  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.115034  9329 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.115558  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.115969  8650 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:48.116400  8659 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:48.110484  8655 raft_consensus.cc:2368] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7.
I20260709 13:02:48.081339  8733 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:48.071686  8651 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 13:02:48.057716 11558 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.111887  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.107738  8654 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:48.023983 11886 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.124825  8579 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:48.127733  8728 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:48.118019  8578 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:48.142129  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:48.148183  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.151665  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:48.153126  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:48.158599  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.160164  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.161201  8652 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:48.145022  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:48.165057  9326 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:48.168725  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:48.081965 11865 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:48.166383  8584 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:48.171372  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.122570 11604 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:48.179708  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:48.187989  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48153.9us
I20260709 13:02:48.191416  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.191727  9327 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:48.191883  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:48.192795  8579 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.179559 11704 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:48.194969 12076 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
I20260709 13:02:48.194592 11704 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.197173 11704 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:48.192190  9323 raft_consensus.cc:2376] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7. Re-sending same reply.
W20260709 13:02:48.197901 12078 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260709 13:02:48.198717  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:48.200114  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.200740  9323 raft_consensus.cc:2376] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6. Re-sending same reply.
I20260709 13:02:48.198858 11871 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:48.201740 11871 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:48.202246 11871 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:48.206283  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.206820 11891 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:48.207553  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.112954 11955 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.193490  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.195665  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.217453  8582 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:48.215173  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.219386  9325 raft_consensus.cc:2410] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:48.220813  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.224556  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:48.220809  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.134920 11558 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.228366  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.185024 11604 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.226970  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.231244  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:48.231570  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.233846  8727 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:48.233130  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.234544  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:48.233093  9329 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:48.235966  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:48.236984  9326 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:48.232724  9323 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:48.240458  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:48.240746  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.240375  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.243647  8578 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:48.213335 11889 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:48.258157  8577 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:48.255503  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:48.263054  9331 raft_consensus.cc:2410] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 13:02:48.264698  8623 leader_election.cc:336] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.794s (SENT)
I20260709 13:02:48.266510  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.269829  9329 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:02:48.270354  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.234398 11604 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:48.276053  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:48.267830  8578 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:48.278687  8728 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:48.233736 12008 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.288353  9325 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:02:48.305059  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.258373 11889 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:48.315789  9328 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:48.231164 12038 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 13:02:48.230654 11865 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.330531  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:48.331459  9324 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.241416 12013 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 13:02:48.351487  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.279619 11900 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:48.353061  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.355645  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:48.374235  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.357972  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.377822  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.334834 11889 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:48.351529 12008 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:48.363193  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 123963us
I20260709 13:02:48.364557  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.370155  8733 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
W20260709 13:02:48.370497  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 143583us
I20260709 13:02:48.282852 11878 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:48.380052  8578 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:48.380826  8728 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:48.382406  8583 raft_consensus.cc:2410] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:48.360925 12038 consensus_queue.cc:1243] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:48.391052  9324 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:48.380409 11900 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.392161 11900 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:48.404618  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:48.405555  8580 raft_consensus.cc:2368] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7.
W20260709 13:02:48.405869  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.407523  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:48.412660  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 336845us
W20260709 13:02:48.413159 12085 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 13:02:48.414839  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:48.415366 12080 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:02:48.416781  8625 leader_election.cc:400] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5.
I20260709 13:02:48.416450 11843 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:48.417248  8625 leader_election.cc:403] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:02:48.418771 11843 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5.
I20260709 13:02:48.419168  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:48.420029  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:48.420802  8733 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.420713  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.423065  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.425895  8584 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:48.425450  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:48.427083  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.428612  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:48.431800 12084 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:02:48.432562 11929 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:48.433135 11929 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.433620 11929 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:48.383580 11865 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.384038 12008 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:48.398339 11878 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.451275  8733 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:48.453589  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.460495  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:48.464375  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104648us
I20260709 13:02:48.450299 11886 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.473608  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.457476 12008 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:48.461581 11878 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:48.483232  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.487370  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:48.490579  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.475000 11569 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 13:02:48.493434  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:48.497985  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.506541  8577 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:48.502214 12038 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:48.533890  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.536161  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:48.539896  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46612.3us
W20260709 13:02:48.538285  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 257181us
I20260709 13:02:48.541500  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:48.545887  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 58614.6us
I20260709 13:02:48.549638  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.550882  8577 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:48.552974  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.500334 11886 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 13:02:48.560514  8728 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:48.522771 12094 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 13:02:48.544400 12013 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:48.569584  8724 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 13:02:48.554792 11704 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:48.586211  8724 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:48.587924 11569 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:48.595237 11569 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:48.596503 11569 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.599272 12096 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:48.599895 12096 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:48.600060  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 185347us
I20260709 13:02:48.600294 12096 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.602380 11886 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:48.603989 12092 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:48.606372 12095 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:02:48.606959 11843 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:48.607529 11843 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:48.615296 11871 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.559944 11900 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.619781 11900 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.678897 11704 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:48.629380 12008 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.679497 11704 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:48.682022 11929 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.682387 11704 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.629272 11955 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.684693 11558 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:48.688602  8579 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:48.690135  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 273644us
I20260709 13:02:48.692399  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:48.608800 11843 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.694924 11604 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.697940  8730 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:48.697190  9328 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:02:48.718236  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 116942us
W20260709 13:02:48.721470  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 157348us
I20260709 13:02:48.718405  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.723670  8577 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:48.732859  9326 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
W20260709 13:02:48.735138  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:48.737752  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.783149  9326 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:48.785789  8728 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:48.788084  8724 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:48.800678 11873 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:02:48.787859  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 246245us
I20260709 13:02:48.746695 11569 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.803923  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:48.802707  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 242274us
W20260709 13:02:48.800275  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 371737us
I20260709 13:02:48.822329  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:48.792013 11843 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:48.826042  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.781585 12038 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:48.834115  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:48.816471 11871 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:48.843421  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:48.848383  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:48.851176  8579 raft_consensus.cc:2410] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:48.829326 11903 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.862310  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2335) took 1082 ms. Trace:
W20260709 13:02:48.864944  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 166167us
I20260709 13:02:48.864667  8726 rpcz_store.cc:276] 0709 13:02:47.779501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.915152 (+135651us) service_pool.cc:224] Handling call
0709 13:02:47.937490 (+ 22338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.980072 (+ 42582us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:47.980114 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:47.980178 (+    64us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:47.980232 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.862034 (+881802us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b5400339854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:48.862089 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.862230 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61403520}
I20260709 13:02:48.885656  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.845631 11604 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.863793 11903 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.899950 12104 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:48.900595 12104 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.901531 11871 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:48.863343 11929 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.904109 11929 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:48.904641 11929 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:48.905241 11903 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:48.905881 12097 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 13:02:48.906608 11878 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.907847  8575 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:48.908459  8581 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:48.908612 12116 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:48.909016 11878 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.909250 12116 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.911897 12116 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:48.909723  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 1997) took 1188 ms. Trace:
W20260709 13:02:48.907928 12086 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 13:02:48.914887  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 81463.1us
W20260709 13:02:48.917088  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 73808.1us
I20260709 13:02:48.917994 11891 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:48.920678  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.913669  8729 rpcz_store.cc:276] 0709 13:02:47.721417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.721572 (+   155us) service_pool.cc:224] Handling call
0709 13:02:47.721895 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:47.843060 (+121165us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:48.273682 (+430622us) log.cc:844] Serialized 14 byte log entry
0709 13:02:48.502910 (+229228us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:48.515968 (+ 13058us) log.cc:844] Serialized 26 byte log entry
0709 13:02:48.516170 (+   202us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:48.516221 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.516312 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:48.909488 (+393176us) raft_consensus.cc:1685] finished
0709 13:02:48.909515 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.909648 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":102180,"prepare.queue_time_us":491165,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":213586,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":4652,"thread_start_us":553376,"threads_started":2,"wal-append.queue_time_us":269526}
I20260709 13:02:48.922716 12104 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:48.854517 11889 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.919629 12013 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:48.899421 12110 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 13:02:48.939167 12123 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 13:02:48.946895 11889 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:48.938118 11938 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:48.948097 11938 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.947813 12116 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:48.948570 11938 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:48.949342  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:48.949534 12116 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:48.950278 12116 consensus_queue.cc:1243] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:48.950646 11938 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:48.950865  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:48.951252 11938 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:48.951618 11938 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:48.951758 12110 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:48.952004 11938 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:48.952273 12110 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:48.952839  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:48.953295 12110 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.956408  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.936093 12038 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:48.963570  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:48.965166  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44631.2us
I20260709 13:02:48.957857 12106 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:48.958788  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:48.965955  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:48.975375  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 108876us
I20260709 13:02:48.940165 11929 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:48.979411  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:48.957590 12107 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:48.984015  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:48.986857  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:48.988894  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:48.974619 11865 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.004519  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.995064  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:48.982086 12013 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:48.938318 11873 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 13:02:48.979877 12099 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:49.009229  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:49.009222  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:49.011628  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:48.987648 11903 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:49.014621  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:49.016871  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 100361us
I20260709 13:02:48.988914 11871 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:49.025036  8732 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 4.
I20260709 13:02:49.024164 11903 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:49.011350 11558 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:49.006680 12106 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:49.043038  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48502.6us
I20260709 13:02:49.023916 11871 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:49.048626 11871 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:49.011765 11865 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:49.051410 11865 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:49.011379 12111 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:49.053727 12103 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:49.053894  8733 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:49.054244 12103 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:49.054710 12098 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 13:02:49.055941 11891 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:49.056535 11891 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:49.056589 12113 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 13:02:49.057458  8727 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:49.054780 11972 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:49.016111 12100 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:49.058357 11972 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:49.036788 11558 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:49.059347 12121 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:49.059868 11558 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:49.059957 12121 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.060539 12121 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:49.054270  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:49.062590 12120 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:49.062839  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:49.063273 11660 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:49.063675 11660 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:49.024698 12099 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.063884 11873 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:49.064754 11873 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:49.064625 11660 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.065222 11873 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.065763  8623 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:49.043803  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 203474us
I20260709 13:02:49.065893 12122 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:49.066022 12099 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:49.066490 12115 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 13:02:49.068367 12124 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:49.068897  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:49.069783  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:49.071959  8696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.108s (SENT) [suppressed 65 similar messages]
I20260709 13:02:49.113250  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.072345 11891 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.075129 12119 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:49.076085  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.076519  8726 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:49.079365  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.083523  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:49.112763  8731 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:49.115037  8657 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:49.117008  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.117707  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.037014 11903 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:49.121201  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.021546 11955 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 13:02:49.037266 12013 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:49.078266 12094 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:49.120688  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.087587 12100 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:49.125849  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54014.2us
I20260709 13:02:49.059974 12110 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.085927 11873 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:49.054718 12103 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.136312  8654 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:49.156011  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:49.137182 12112 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:49.141860  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:49.130919 12013 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:49.168793  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:49.139621  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:49.141564  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 151550us
W20260709 13:02:49.146309  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.170492  8657 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.160162 11569 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:49.177767 12100 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:49.179524  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:49.180217  8659 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:49.164338 12114 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:49.177667  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:49.188090  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 122351us
I20260709 13:02:49.059363 11972 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:49.193219  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 169137us
I20260709 13:02:49.204614  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:49.211475  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 130821us
W20260709 13:02:49.215750  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.218194  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 146850us
I20260709 13:02:49.184430 12114 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:49.171664 12013 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:49.212407 12096 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.612s	user 0.010s	sys 0.003s
I20260709 13:02:49.232364  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:49.239017  8581 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
W20260709 13:02:49.233312  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60683.8us
I20260709 13:02:49.239893  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.190922 11886 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:02:49.256611  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 490) took 1227 ms. Trace:
I20260709 13:02:49.226679 12114 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:49.153252 12121 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.277879  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.278782  8650 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:49.286827  8659 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:49.169760 12106 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:49.290343  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 121635us
I20260709 13:02:49.263351 11886 consensus_queue.cc:1243] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:02:49.292608  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 88095.9us
I20260709 13:02:49.253511 12122 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:49.300357  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:49.293658  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:49.297957  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 102795us
I20260709 13:02:49.299196  8728 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:49.301076  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.262681  8656 rpcz_store.cc:276] 0709 13:02:48.014165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.016395 (+  2230us) service_pool.cc:224] Handling call
0709 13:02:48.026047 (+  9652us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:48.137499 (+111452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:48.137549 (+    50us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:48.533788 (+396239us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b2c000b1fc8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:02:48.542985 (+  9197us) log.cc:844] Serialized 78 byte log entry
0709 13:02:48.747053 (+204068us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:48.889784 (+142731us) log.cc:844] Serialized 14 byte log entry
0709 13:02:48.924349 (+ 34565us) log.cc:844] Serialized 14 byte log entry
0709 13:02:49.085301 (+160952us) log.cc:844] Serialized 14 byte log entry
0709 13:02:49.231510 (+146209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.241038 (+  9528us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:49.241086 (+    48us) raft_consensus.cc:1685] finished
0709 13:02:49.241098 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:49.241487 (+   389us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46563,"prepare.queue_time_us":373040,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":2883,"raft.queue_time_us":217077,"raft.run_cpu_time_us":564,"raft.run_wall_time_us":32182,"spinlock_wait_cycles":29733248,"thread_start_us":594527,"threads_started":5,"wal-append.queue_time_us":268741}
I20260709 13:02:49.301945  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:49.302902  8659 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:49.310276  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.311122  8654 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:49.247375 12096 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:49.320629  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 89531.2us
I20260709 13:02:49.292102  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:49.323719  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:49.324589  8653 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:49.286144 12108 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.181s
I20260709 13:02:49.329219  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.319672 12099 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:49.299022 11938 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.319204 12130 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:02:49.322971  8549 connection.cc:256] RPC call timeout handler was delayed by -1.957225606999998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
I20260709 13:02:49.357445  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.328565 12127 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:49.352129  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:49.355881  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.356628  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.327993 12115 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:49.358685  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66634.9us
I20260709 13:02:49.367313  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:49.370685  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:49.379135  8659 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:02:49.360759 11891 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:49.380602  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.380952 12094 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:49.381507 12094 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:49.381750 12141 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.382326 12141 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.384691 12141 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:49.384807 11660 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:49.386586 12133 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:02:49.380651  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:49.392828  8727 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:49.365486 11865 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.395730  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:49.395114 12108 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
W20260709 13:02:49.393605  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.397475 12111 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:49.393389 11871 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.397835 12140 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 13:02:49.407325 11886 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:49.408209 11886 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:49.408368  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:49.408569 12140 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:49.409598 12127 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:49.398247  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:49.410084 12128 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:49.410454  8582 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:49.399843 11569 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:49.405138 11878 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:49.456177  8549 connection.cc:256] RPC call timeout handler was delayed by -1.4924177999999984s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17250000000000001s for a response
I20260709 13:02:49.405649 12121 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:49.406070 11704 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:49.406620 12115 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:49.408705 11900 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:49.412503  8657 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:49.404338  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:49.418187  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:49.428139  8653 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:49.372080 12129 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 13:02:49.394218 12131 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:02:49.456676  8549 connection.cc:256] RPC call timeout handler was delayed by -1.3403177999999798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20260709 13:02:49.461771  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 142695us
I20260709 13:02:49.464926  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:49.385241 12141 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.396832 11955 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:49.332518 12132 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.103s
I20260709 13:02:49.434883 11569 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:49.480777  8730 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 13:02:49.492195  8549 connection.cc:256] RPC call timeout handler was delayed by -1.2170178000000078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20260709 13:02:49.503010  8549 connection.cc:256] RPC call timeout handler was delayed by -1.1648177999999803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
I20260709 13:02:49.436372 12136 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.408223 12008 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:49.481411  8652 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:49.491293  8581 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.456391 11878 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:49.502691  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:49.473717  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 106591us
I20260709 13:02:49.468786 11891 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:49.435904 11558 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:49.468436 12124 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 13:02:49.514293 11797 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:49.509684  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 91544.3us
I20260709 13:02:49.540634  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:49.513652 11929 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.007s
W20260709 13:02:49.505035 11843 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.671s	user 0.011s	sys 0.002s
W20260709 13:02:49.524416  8696 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.711s (SENT)
I20260709 13:02:49.494360 11900 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:49.552918  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.546803  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.555159  8653 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:49.547895  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.548645  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.516917 12111 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:49.550390  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:49.560222  8659 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:49.543789  9296 leader_election.cc:400] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7.
I20260709 13:02:49.560977  9296 leader_election.cc:403] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:49.554404  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:49.554590  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.521337 11891 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:49.549732  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:49.549559  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:49.557664  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:49.375977 12122 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.559099  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:49.501252 11955 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:49.563637  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:49.495139 12141 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.565317  8656 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:49.584684  9323 raft_consensus.cc:2368] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7.
I20260709 13:02:49.585281  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.577450  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:49.566356  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:49.525154 12139 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:49.569566  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:49.539975 11797 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:49.598424 12156 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7.
I20260709 13:02:49.598637  8579 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:49.599175  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:49.574438  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:49.581339 11871 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:49.581485 12096 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:49.581153 12111 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:49.582506  8733 pending_rounds.cc:85] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:02:49.677796  8733 pending_rounds.cc:107] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:49.678194  8733 raft_consensus.cc:2889] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:49.552464 12128 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:49.588675  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:49.588828  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64429.4us
W20260709 13:02:49.679850  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.907s (SENT)
I20260709 13:02:49.680433  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:49.680763  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 103364us
I20260709 13:02:49.591401  8581 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:49.582773 11558 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:49.598042 11900 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.602695  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:49.578758 11569 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:49.598141  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:49.606746  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 661) took 1188 ms. Trace:
I20260709 13:02:49.671528  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:49.543027 12147 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.154s
I20260709 13:02:49.568017 11891 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:49.563150 12110 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:49.674450  9329 pending_rounds.cc:85] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 6
W20260709 13:02:49.675765  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 144233us
I20260709 13:02:49.733451  9329 pending_rounds.cc:107] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 13:02:49.546197 12114 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.542455 11873 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.515993 12136 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.589205 11886 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:49.690176  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 87535.1us
I20260709 13:02:49.559559 12123 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.113s
W20260709 13:02:49.672781  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70559.7us
I20260709 13:02:49.704531  8583 rpcz_store.cc:276] 0709 13:02:48.401972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.402272 (+   300us) service_pool.cc:224] Handling call
0709 13:02:48.438213 (+ 35941us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:48.831977 (+393764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:48.832023 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:48.838596 (+  6573us) log.cc:844] Serialized 26 byte log entry
0709 13:02:49.039429 (+200833us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:49.039483 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.039577 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:49.589928 (+550351us) raft_consensus.cc:1685] finished
0709 13:02:49.589947 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:49.590063 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":160114,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":518092,"threads_started":2,"wal-append.queue_time_us":236201}
W20260709 13:02:49.713704  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 147725us
I20260709 13:02:49.602622 11955 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:49.606449 12038 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:49.729506 11660 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:49.733749  9329 raft_consensus.cc:2889] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:49.693557 12100 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:49.734400  8698 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.864s (SENT)
I20260709 13:02:49.702704 12008 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:49.730329 12137 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:49.741139  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 200905us
I20260709 13:02:49.743625  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:49.724969  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45165.2us
I20260709 13:02:49.739399 12110 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:49.598466 11797 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.761950  8575 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:49.771469  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:49.747428 11900 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:49.749852 12123 consensus_queue.cc:1243] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:49.764639  8622 leader_election.cc:400] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7.
I20260709 13:02:49.792528  8622 leader_election.cc:403] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:02:49.786890  9328 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:49.752545 12038 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.751421 11660 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.798674  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:49.604102 12115 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.752835 12100 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:49.751055 12108 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:49.809267  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44740.6us
I20260709 13:02:49.810585  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:49.714690 11878 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:49.770120 12110 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:49.812175  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.812142  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:49.785494 12131 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:49.790302 11569 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:49.800926 11660 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:49.819110 11797 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:49.801586 12038 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:49.820559  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:49.824894  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 90912.3us
I20260709 13:02:49.809417 12100 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.820708 11886 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7.
I20260709 13:02:49.822872 12145 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:49.878803  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65749.8us
I20260709 13:02:49.879933  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:49.824221 12157 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:02:49.864742 12150 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.304s	user 0.001s	sys 0.000s
W20260709 13:02:49.897112  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 84955.1us
W20260709 13:02:49.874511 12142 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 13:02:49.916563  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:49.918920  8582 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.508s	user 0.009s	sys 0.002s
I20260709 13:02:49.822620 12108 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.909730 11878 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:49.910146 11903 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.756s	user 0.011s	sys 0.000s
W20260709 13:02:49.910733 12153 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.225s	user 0.001s	sys 0.000s
W20260709 13:02:49.912170 12149 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20260709 13:02:49.858321 12158 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:49.919503  8582 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:49.913952 11873 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:49.930411  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.926142  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50677.4us
I20260709 13:02:49.932020 12128 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:49.926594 12131 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:49.928326  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:49.943681  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:49.947167  9327 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:49.923019 12123 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:49.954409 12131 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.955571  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:49.956568  8696 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.893s (SENT)
I20260709 13:02:49.932989 12141 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:49.935420 11903 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.957707  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:49.957825 11886 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:49.958647 12152 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260709 13:02:49.959116 12166 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:49.963137 12108 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:49.963425 12155 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20260709 13:02:49.965911 12148 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260709 13:02:49.970993 12179 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:49.972711  8651 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:49.976919  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:49.957197  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:49.933331 12156 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:50.013463 12159 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
W20260709 13:02:50.013886  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57339.7us
I20260709 13:02:50.014788 12123 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:50.015743 12136 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:50.016520 12128 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:49.994375  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:49.932797 12160 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
W20260709 13:02:49.983731 12103 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.814s	user 0.006s	sys 0.006s
I20260709 13:02:50.027221 12103 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.028894 12179 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.031543 12179 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:50.003173  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:49.989863 12094 consensus_meta.cc:287] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.608s	user 0.011s	sys 0.001s
I20260709 13:02:50.007212 12158 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:50.007383 12164 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:50.007999 12143 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:50.008293  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51167.7us
I20260709 13:02:50.045077  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.046778  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.048251  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:49.999933 12166 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:50.022136  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.021961  8656 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:49.951439 11972 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:02:49.997180 12114 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:50.022356  9331 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:50.050705  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:50.006507 11865 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.605s	user 0.005s	sys 0.005s
I20260709 13:02:50.052384  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.058995  9322 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:50.060387  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.061245 11903 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:50.063910  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65726.2us
W20260709 13:02:50.067466  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45407.9us
I20260709 13:02:50.061336  9331 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:50.071673  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:50.079119  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:50.080049  8657 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:50.081665 11604 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:50.082492  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:50.082297 11604 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.084503 11604 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:50.084755  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:50.086759  8655 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:50.085500  8659 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:50.090350  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.081195  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:50.054472 12179 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:50.108330  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:50.114811  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:50.115527  8655 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:50.117182  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:50.118321  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.118000  9325 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:50.057243 12166 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:50.076551 12092 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:50.073859 11865 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:50.039840 12094 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.117832 11903 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:50.085465 11891 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.111531 12179 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:50.133672  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:50.128306 12013 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.874s	user 0.012s	sys 0.000s
I20260709 13:02:50.113158 12160 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:02:50.139510  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.125936 12092 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:50.128155 12132 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:50.134197 11900 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:50.140610  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:50.138173 12179 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:50.158022  9329 raft_consensus.cc:2376] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8. Re-sending same reply.
I20260709 13:02:50.158495  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:50.158351 12092 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:50.181932 12175 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:02:50.140334 12121 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.006s	sys 0.022s
W20260709 13:02:50.229537 12121 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.007s	sys 0.023s
I20260709 13:02:50.139573 12181 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:50.186057 12175 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.141300 12013 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.135018 11660 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:50.193810  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55807.8us
I20260709 13:02:50.185302 12132 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.172891 11972 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:50.201905  9331 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.209038  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:50.194242 12101 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.210238  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:50.210759 12110 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.211594  8730 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.212368  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:50.212553 12094 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:50.213145 12099 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.853s	user 0.017s	sys 0.000s
I20260709 13:02:50.215286 12038 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.218534  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:50.224807 11886 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:50.290496 11660 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:50.290915 11886 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:50.291344 12038 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:50.291903 11886 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.232816  9322 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:50.239605  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:50.293771  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54241.7us
I20260709 13:02:50.293952  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.294626  9322 raft_consensus.cc:2376] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 7. Re-sending same reply.
I20260709 13:02:50.294756  9328 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:50.295658  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.296336 11955 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:50.296916  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:50.278659  9329 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:50.306749  9328 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:02:50.279685  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:50.307613  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.281838  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:50.284066  9326 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.283991  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.287417  8583 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:02:50.245853 12181 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:50.292554  9330 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:50.297744  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 663) took 1081 ms. Trace:
I20260709 13:02:50.298236  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:50.317094  8575 rpcz_store.cc:276] 0709 13:02:49.215927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.216076 (+   149us) service_pool.cc:224] Handling call
0709 13:02:49.229084 (+ 13008us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.768331 (+539247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.297453 (+529122us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b54001685d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:50.297678 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1224597760}
I20260709 13:02:50.297137 11955 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:50.303222 12110 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:50.304219 12099 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:50.297751  8696 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.013s (SENT)
I20260709 13:02:50.322521  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.322919 12108 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:50.323352 12108 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.323691 12108 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:50.324390 12111 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:50.325616 12119 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:50.326021 12119 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:50.326936 12119 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.328132 12119 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:50.331192 12013 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:50.305708  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:50.335180  8577 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:50.274863  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:50.237655  8655 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:50.306384 11886 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:50.270359 12175 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:50.276968 11972 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:50.319445  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:50.246873 12100 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.305924  9322 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:50.309422  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:50.337805  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:50.316663 12181 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:50.342896  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:50.346860 11797 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:50.347620 11955 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:50.349884  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51723us
I20260709 13:02:50.346380  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:50.288437 12101 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.365288 12101 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:50.370200 11938 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:50.373018 12132 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:50.375298 12100 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:50.376253  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:50.376389  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.378230  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.379148 12112 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:50.380748 12112 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:50.382253  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.384538 12094 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:50.388458 11569 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.389292 11569 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.345021 11972 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:50.385149  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:50.356185 12013 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:50.408974  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:50.361869 12104 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:50.416072 11891 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:50.318969 12092 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.422559  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.427299  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:50.421298 12193 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 13:02:50.413187 12112 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:50.442210  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:50.425519 11797 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:50.426883 11569 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:50.432652 12114 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:50.430639 12013 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:50.437841 12128 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.452575 11797 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:50.452512 12128 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.441360 12112 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:50.426728 12166 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.447034 11878 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:50.445693 12104 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:02:50.453722 11797 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.460634 12114 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:50.446130 12179 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.474141 12112 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:50.480476 12100 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.648133 11878 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:50.661339  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 238801us
I20260709 13:02:50.650697 11873 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.652429 12099 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:50.650391  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:50.668326  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.687086  8582 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:50.452688 12013 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.656560 12177 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:50.665974 11878 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:50.699128  9323 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.701470 12168 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.701880 12092 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:50.707414  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.713001  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:50.718052  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.721161  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:50.723557  9323 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:50.728250  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.732336  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.667677 12100 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.759270  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:50.702085 12168 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.742794  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:50.763181  9323 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:50.764904  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.765743  9329 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:50.775308  9328 pending_rounds.cc:85] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 5
I20260709 13:02:50.775714  9328 pending_rounds.cc:107] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 13:02:50.775942  9328 raft_consensus.cc:2889] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:50.692222 12099 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:50.780018  8623 leader_election.cc:400] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7.
I20260709 13:02:50.689594 11873 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:50.783030  9323 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 13:02:50.798633  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66571us
I20260709 13:02:50.740671 12092 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:50.726076 12128 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:50.757779 12163 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:50.779023 12140 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:50.800326  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:50.804149  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 139384us
I20260709 13:02:50.818488  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.761880 12104 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:50.807417 12099 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:50.807830 12201 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
W20260709 13:02:50.811899 12194 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
I20260709 13:02:50.819109  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:50.793753 11955 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:50.823123 12092 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:50.817379 12114 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:50.839097  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.814469 12163 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:50.851814  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.534s (SENT)
I20260709 13:02:50.852656  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:50.854715  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:50.855616  8655 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:50.856779  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:50.858196  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:50.858987  8656 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:50.862036  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:50.863070  8655 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:50.925557  9329 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:50.939692  8696 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.864s (SENT)
I20260709 13:02:50.940435  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.926697  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:50.922438  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:50.973387  8727 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.977535  8652 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:50.978312  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 53841us
I20260709 13:02:50.979351  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:50.982343  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:50.983705  8659 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:50.982995  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:50.988238  9329 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:50.991446  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:50.993060  8658 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: [4.3-5.4]
I20260709 13:02:50.858510 12163 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:50.999923  8655 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:50.829368 12160 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.136s
I20260709 13:02:51.009795  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:51.018729  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:50.868497 12140 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:51.045065  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:51.046034  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 106372us
I20260709 13:02:50.928508 12110 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:50.857999 12092 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:51.058355  8656 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:51.052551  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:51.066486  8653 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:51.068733  8727 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:51.069221  8727 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:51.073575  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:51.051332 12110 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.079360 12092 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:51.083329 12094 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.698s	user 0.010s	sys 0.000s
I20260709 13:02:51.010181 11891 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:51.047417 11900 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:51.127154  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 308670us
I20260709 13:02:50.994500 12166 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:51.064540 12180 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 13:02:51.064815 12153 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:02:51.080320 12110 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:51.151242  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 228838us
I20260709 13:02:51.050350 12140 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.063925 12013 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:51.147190  8727 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.085232 12154 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:51.170534  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:51.174726  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:51.086182 12179 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:51.176699 12119 consensus_meta.cc:287] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.848s	user 0.013s	sys 0.001s
I20260709 13:02:51.157672 12201 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:51.135682 12094 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:02:51.214501  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 223065us
I20260709 13:02:51.211714  8727 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:51.200464 12108 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.876s	user 0.009s	sys 0.000s
I20260709 13:02:51.164505 11938 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:51.190928 12197 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.481s	user 0.001s	sys 0.001s
W20260709 13:02:51.218719  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44058.2us
I20260709 13:02:51.222481  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:51.224813  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:02:51.197342 12198 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.385s	user 0.000s	sys 0.001s
W20260709 13:02:51.219863  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92466.2us
I20260709 13:02:51.234587  9329 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 13:02:51.235051  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 252083us
I20260709 13:02:51.208331 12140 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:51.254894  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:51.201887 11900 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 13:02:51.237007 12013 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:51.238539 12177 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:51.248039 12201 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:51.234637 12213 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:51.301580  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.303458  8584 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:51.311082 11886 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.969s	user 0.008s	sys 0.001s
I20260709 13:02:51.312085  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.338415  8584 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
W20260709 13:02:51.313118 12202 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260709 13:02:51.272171 12094 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:51.257867 11873 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:51.319213 12184 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 1.194s	user 0.000s	sys 0.004s
I20260709 13:02:51.321843  8655 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:51.326004 12214 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:02:51.325975  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.327059  8579 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:51.327878  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.356098  8581 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:51.328802  8577 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:51.329694 12206 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 13:02:51.247397 12168 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:51.331503 12212 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
W20260709 13:02:51.332446 12203 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.476s	user 0.001s	sys 0.000s
I20260709 13:02:51.332969  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.280843 12013 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:51.334074 12205 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 13:02:51.334494 11900 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:51.350982  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.346031 12213 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:51.238688 11938 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.330209 12216 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:51.380386 12216 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:51.380860 12216 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:51.237668 12108 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:51.387301 12108 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.362533  8576 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:51.359668 12094 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:51.389668  8578 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:51.390364 12094 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:51.392648  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.376076 11900 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:51.394325 11900 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:51.395669 11900 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:51.398473 11878 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.690s	user 0.010s	sys 0.000s
I20260709 13:02:51.399032 11878 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.377924  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.403496  8577 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:51.276081 12100 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:51.360193 11972 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.969s	user 0.007s	sys 0.004s
I20260709 13:02:51.376945  8727 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:51.377741  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.378336  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.415362  8581 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:51.407009  8578 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
W20260709 13:02:51.409677  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:51.385169 12181 consensus_meta.cc:287] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.022s	user 0.009s	sys 0.002s
I20260709 13:02:51.413846  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.419581  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.461880  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.406629 11878 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:51.420619 11938 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:51.472080 12104 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.649s	user 0.008s	sys 0.001s
I20260709 13:02:51.472584 12104 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:51.479583  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.482786  8578 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:51.462443 12175 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.103s	user 0.012s	sys 0.000s
I20260709 13:02:51.484447 12175 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.409683 12213 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.487274 12213 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:51.421984  8579 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:51.490716 12181 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.497733 12175 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:51.505939  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.506897  8581 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:02:51.421595 11972 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.508853  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:51.508718 12112 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.839s	user 0.012s	sys 0.000s
I20260709 13:02:51.519204  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.523312  8728 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:51.527215  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 670) took 1296 ms. Trace:
I20260709 13:02:51.527805  8582 rpcz_store.cc:276] 0709 13:02:50.230441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.230689 (+   248us) service_pool.cc:224] Handling call
0709 13:02:50.231104 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.692588 (+461484us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.526857 (+834269us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b540002e754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:51.527125 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1013793152}
W20260709 13:02:51.538141 12191 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.174s	user 0.007s	sys 0.003s
I20260709 13:02:51.539681  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 651) took 1525 ms. Trace:
I20260709 13:02:51.540062  9327 rpcz_store.cc:276] 0709 13:02:50.014617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.017492 (+  2875us) service_pool.cc:224] Handling call
0709 13:02:50.017864 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:50.018310 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.018337 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:50.019019 (+   682us) log.cc:844] Serialized 26 byte log entry
0709 13:02:50.308986 (+289967us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:50.309024 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.309101 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:51.537677 (+1228576us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b24000c5318. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:02:51.539326 (+  1649us) raft_consensus.cc:1685] finished
0709 13:02:51.539350 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:51.539592 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7242,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":95534848,"thread_start_us":289558,"threads_started":1,"wal-append.queue_time_us":334657}
I20260709 13:02:51.541738 12013 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.543942 11972 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:51.544353  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2362) took 1337 ms. Trace:
I20260709 13:02:51.544816  8732 rpcz_store.cc:276] 0709 13:02:50.206589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.208092 (+  1503us) service_pool.cc:224] Handling call
0709 13:02:50.208438 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.305624 (+ 97186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.305656 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.305699 (+    43us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:50.305731 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.544071 (+1238340us) spinlock_profiling.cc:243] Waited 97 ms on lock 0x7b54004218d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:51.544126 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:51.544265 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223001472}
I20260709 13:02:51.501219 12140 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.548672  8579 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:51.559522  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.563686  8579 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:51.565840  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.576843  8581 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:51.578478  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.579412  8579 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:51.580255  8577 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:51.583029  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.583770  8579 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:51.584635  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.585521  8577 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:51.588717  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.589840  8581 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:51.578899 12163 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260709 13:02:51.595849  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:51.596521  8650 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:51.597306  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:51.522650 12112 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:51.598127  8656 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 5: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:51.578400 12127 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.602141  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.520068 12216 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:51.597450  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:51.611497  8652 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:51.622368  8580 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:51.622654  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:51.623579  8657 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:51.613890 12127 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:51.630576  8575 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:51.607280 12103 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.618043  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.636238  8579 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:51.638814  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:51.642493  8581 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:51.650682  8577 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:51.653461  8581 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:02:51.659462  8582 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:51.654453  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:51.660625  8658 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:51.609130 12119 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.026s
I20260709 13:02:51.620639 12175 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:51.605906 12163 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.675153  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.678997  8579 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:51.680096 12181 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:51.682308 12216 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.631987 12130 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.631817 12143 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.631943 11938 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:51.690248  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.694262  8658 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:02:51.690925  8576 raft_consensus.cc:2376] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6. Re-sending same reply.
I20260709 13:02:51.696696  8652 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:51.669528 12103 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:51.688953  8650 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:51.710167  8652 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:51.717163  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.720975  8579 raft_consensus.cc:2376] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8. Re-sending same reply.
W20260709 13:02:51.702415 11955 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.069s	sys 0.118s
I20260709 13:02:51.689790 12175 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:51.724848 11955 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.069s	sys 0.118s
I20260709 13:02:51.717099 12103 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:51.752210  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 861) took 1356 ms. Trace:
I20260709 13:02:51.837173  9322 rpcz_store.cc:276] 0709 13:02:50.395664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.395811 (+   147us) service_pool.cc:224] Handling call
0709 13:02:50.411440 (+ 15629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.411870 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:50.426588 (+ 14718us) log.cc:844] Serialized 14 byte log entry
0709 13:02:51.531458 (+1104870us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b18003be540. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:02:51.751686 (+220228us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:51.751776 (+    90us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:51.751848 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.751951 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:51.752125 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13760,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":1438,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":100,"spinlock_wait_cycles":5501696,"thread_start_us":217003,"threads_started":1}
W20260709 13:02:51.837139  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 186017us
I20260709 13:02:51.841034  8653 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 13:02:51.705950 12130 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.706094 12143 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.707466 11938 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.858251  8584 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:51.859155  8583 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:51.849871  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2328) took 2762 ms (client timeout 2350 ms). Trace:
I20260709 13:02:51.720613 12229 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.136s
I20260709 13:02:51.852285 11891 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:51.853577  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.867888  8579 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:51.864750 12143 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.841616 12140 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:51.873354  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 877) took 1158 ms. Trace:
W20260709 13:02:51.875608  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 235141us
W20260709 13:02:51.865998  8651 rpcz_store.cc:269] 0709 13:02:49.077971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.078130 (+   159us) service_pool.cc:224] Handling call
0709 13:02:51.840356 (+2762226us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54001d1ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:51.840491 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1926627712}
I20260709 13:02:51.875438  9328 rpcz_store.cc:276] 0709 13:02:50.714969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.715107 (+   138us) service_pool.cc:224] Handling call
0709 13:02:50.775083 (+ 59976us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:50.794012 (+ 18929us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.794042 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:50.799722 (+  5680us) log.cc:844] Serialized 52 byte log entry
0709 13:02:51.538166 (+738444us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b1800352c60. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:02:51.749955 (+211789us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:51.820996 (+ 71041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.821118 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:51.867112 (+ 45994us) raft_consensus.cc:1685] finished
0709 13:02:51.867135 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:51.873153 (+  6018us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1090,"raft.queue_time_us":1663,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":10498,"spinlock_wait_cycles":15108992,"thread_start_us":211526,"threads_started":1,"wal-append.queue_time_us":1052377}
I20260709 13:02:51.879988  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.878652 12103 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.883891  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:51.890452  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.893335 12175 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.895828 12163 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:51.895978  8578 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:51.882179 12112 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.901365  8584 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:51.903009  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.903012  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:51.903836  8584 raft_consensus.cc:2376] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5. Re-sending same reply.
I20260709 13:02:51.904783  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.905345  8581 raft_consensus.cc:2376] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8. Re-sending same reply.
I20260709 13:02:51.906909  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.898689 11891 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:51.916433  8576 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:51.933928  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.937234  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:51.952384  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.932299 12140 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.964756  8583 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:51.919631 11891 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:51.973104  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.974042 12237 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:02:51.974334 12234 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:02:51.976044  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:51.980610  8650 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:51.981665  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:51.977762  8582 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:51.988586  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:51.990429  8584 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:51.990223  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:51.992659  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:51.995656 11938 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:51.995716 12094 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:51.996671 12094 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:51.997555  8652 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:51.998020  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 870) took 2428 ms. Trace:
I20260709 13:02:51.998643  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:51.998966 12094 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:51.999750 12094 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.000240 12094 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.002698 12094 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:52.003331  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 958114us
I20260709 13:02:52.003973 11865 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:52.004562 11865 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.004784 12154 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:52.005877 12154 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:52.005776  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 755584us
I20260709 13:02:52.006479 12154 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.007445 12154 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.008569 12154 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.003448 12215 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:52.009905 12215 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:52.010418 12215 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.010344 12219 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:52.004787  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:52.012881  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:52.003983  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.013664 12158 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:02:52.014380  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.010033 12092 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 13:02:52.017575  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2034) took 1662 ms. Trace:
I20260709 13:02:52.017741  8732 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:52.003298  8549 connection.cc:256] RPC call timeout handler was delayed by -1.0288732819999495s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23500000000000001s for a response
I20260709 13:02:52.007494  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2361) took 1801 ms. Trace:
I20260709 13:02:52.022718  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.003600  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2035) took 1643 ms. Trace:
I20260709 13:02:51.975649 12143 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.020193  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:52.034493  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:51.972834 12140 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.042382  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.043004  8724 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:52.043779  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:52.044411  8724 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:52.036752  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.045801  8650 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:51.998442  9330 rpcz_store.cc:276] 0709 13:02:49.569484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.569641 (+   157us) service_pool.cc:224] Handling call
0709 13:02:51.997819 (+2428178us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b5400319054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:51.997960 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1592236288}
I20260709 13:02:52.023121  8728 rpcz_store.cc:276] 0709 13:02:50.206086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.212893 (+  6807us) service_pool.cc:224] Handling call
0709 13:02:50.213334 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:51.545556 (+1332222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.007241 (+461685us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400065254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:52.007420 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3011558656}
I20260709 13:02:52.039379  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.040359  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.048108  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.007499  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:52.049263  8727 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:52.005671 11865 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.050522  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.018051  8725 rpcz_store.cc:276] 0709 13:02:50.355003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.813003 (+458000us) service_pool.cc:224] Handling call
0709 13:02:50.825830 (+ 12827us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.826457 (+   627us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:50.859315 (+ 32858us) log.cc:844] Serialized 14 byte log entry
0709 13:02:51.346687 (+487372us) log.cc:844] Serialized 14 byte log entry
0709 13:02:51.533203 (+186516us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b1800433cc0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:52.017192 (+483989us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:52.017261 (+    69us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:52.017309 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.017366 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.017515 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1131891,"prepare.queue_time_us":30551,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":116,"raft.queue_time_us":802845,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":237,"spinlock_wait_cycles":42837760,"thread_start_us":478501,"threads_started":1}
I20260709 13:02:52.050638 11900 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.050695 11938 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:52.052138 11797 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:52.052549 12112 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:52.052764 12181 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.054005 12209 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:52.055245  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.055722  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:52.080390  8727 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.057075  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.082921  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:52.047279  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.084224  8728 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
W20260709 13:02:52.061650  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2471) took 2512 ms (client timeout 1761 ms). Trace:
I20260709 13:02:52.059700  8583 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:52.064779  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:52.061755  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54347.3us
I20260709 13:02:52.064939  8652 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
W20260709 13:02:52.085464  8654 rpcz_store.cc:269] 0709 13:02:49.549341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.549511 (+   170us) service_pool.cc:224] Handling call
0709 13:02:52.059985 (+2510474us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b540016e4d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:52.061583 (+  1598us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1586493312}
I20260709 13:02:52.063755  8725 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.086404  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:52.087663  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.087869  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.090888  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.065654  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.024868  8726 rpcz_store.cc:276] 0709 13:02:50.359952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.849794 (+489842us) service_pool.cc:224] Handling call
0709 13:02:50.850123 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.850328 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:50.858776 (+  8448us) log.cc:844] Serialized 14 byte log entry
0709 13:02:51.013788 (+155012us) log.cc:844] Serialized 14 byte log entry
0709 13:02:52.003052 (+989264us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b1800433d20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:02:52.003215 (+   163us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:52.003267 (+    52us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:52.003325 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.003398 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.003543 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":202820,"prepare.queue_time_us":143815,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":17455,"raft.queue_time_us":1112824,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":17884,"spinlock_wait_cycles":37880704,"thread_start_us":149672,"threads_started":1}
I20260709 13:02:52.061798  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.092160  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.056499 12220 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.069921  8650 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:52.069969  8730 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:52.066042 11797 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.071451 11938 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:02:52.072119 12172 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.909s	user 0.005s	sys 0.019s
I20260709 13:02:52.059543 11604 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.072574 12140 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.059538 12175 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 13:02:52.076242  8651 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:52.077282 12181 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.096201  8729 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:52.092926  8726 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
W20260709 13:02:52.097853  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 78348.6us
I20260709 13:02:52.102097  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.102984  8579 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:52.109344  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.136541  8724 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 4.
I20260709 13:02:52.114171  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.104117  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.063225 12092 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:52.119534  8727 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:51.987457 12237 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:52.133430 12110 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.942s	user 0.014s	sys 0.000s
I20260709 13:02:52.149246  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.149945  8728 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:52.145428  8696 leader_election.cc:336] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.832s (SENT)
I20260709 13:02:52.079200 12209 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.148221  8650 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:52.103278 12163 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:52.103396 11929 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:52.152616  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:52.163967  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 100425us
I20260709 13:02:52.114073 12099 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.114284 11938 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:52.074607 11900 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.166553  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:52.167150  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.168105  9329 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:52.058558 12143 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.169162  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:52.174149  8696 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.108s (SENT)
I20260709 13:02:52.174431  9330 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.174748  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:52.095165 12219 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 13:02:52.178783  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:52.172710  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.179391  9329 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.173388  8732 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:52.155031 12112 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.145337 11955 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.090420 11891 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.197901  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.165000 11929 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.199261 11929 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.199632  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.200498 11929 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.159260 12110 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.183209  9328 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:52.201918  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:52.202517  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.202874  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:52.201207  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.203799  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.204609  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.204699  9328 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:52.205626  8653 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:52.186497  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:52.160270 12092 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.188186  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.211051  9330 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:52.179803  8584 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:52.212715  9329 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:52.184319  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.192898  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.193977 12111 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.192795 11797 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.194453 11558 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:52.217764 12232 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 13:02:52.218330 11900 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.220189  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.220871  8732 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:52.196079 12219 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:52.197032  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.159766 12236 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 13:02:52.171820  8579 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:52.182529  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.204615  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.208242  9324 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:52.121448 12220 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.135555 11604 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.176887 12099 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.222940 12219 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.236145  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.237118 12219 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.238675 12219 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.191371  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.242403  8581 raft_consensus.cc:2376] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5. Re-sending same reply.
I20260709 13:02:52.152052 12228 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 13:02:52.243515 12245 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:52.244480 12094 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:52.244544 12245 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.245174 12094 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.245352 12245 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:52.246371 12094 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.247831 12094 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.243598  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.250415  8581 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:52.239599  8654 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.240485  8651 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.214412  8650 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:52.240485  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 86409.6us
I20260709 13:02:52.217150 11797 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.203001 12110 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.270388  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.184371 12130 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.277611  8652 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:52.278677  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:52.282841  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 75652.2us
I20260709 13:02:52.286945  8579 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:52.288300  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.289014  8732 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:52.297602  8650 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:02:52.301896  8581 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:02:52.303726  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.206549 12112 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.310482  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.320892  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:52.290225 12229 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:52.324292 12229 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:52.328272  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.337236  8579 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.310441 11900 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.339154  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.288501 12171 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
W20260709 13:02:52.350975  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 109741us
I20260709 13:02:52.326150 12229 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.354964 11558 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.355751 11558 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.310041 12111 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 13:02:52.336452 12215 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.358201 12252 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.358958 12175 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:52.358832 12252 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.359417 12228 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.359948 12215 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:52.360739  8731 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:52.361159  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.361534 12252 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.362519  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:52.349828  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 147000us
I20260709 13:02:52.350454 12143 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:52.311942 12163 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.369465  8583 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:52.378132  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.370417  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:52.291296 12248 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 13:02:52.374696 12181 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.299431 11938 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.401552  8583 raft_consensus.cc:2368] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6.
W20260709 13:02:52.385135  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46096.2us
I20260709 13:02:52.335680 11865 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 13:02:52.371013 11900 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.372120 12215 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:52.393584 12143 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:52.394676 12163 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.371477 12140 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.382653  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.423066  8731 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:52.404181  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.406651 12253 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:52.396768 12245 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.409622  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.412462  8651 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:52.407675 12243 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 13:02:52.423098  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 102316us
I20260709 13:02:52.369568 12228 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.407650  8579 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:52.429019  8654 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:52.427358  8583 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:52.322273 12092 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.439356  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.440155  8583 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:52.414156 12154 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:52.441041  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:52.446310  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:52.447005  8583 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.425153 12215 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.430050 12130 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.459040  8725 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:02:52.438501 12143 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:52.443473 12154 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.444079 11938 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.426438 12140 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.442170 12181 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.472560  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.455868 12111 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.476720  8725 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:52.457131 12215 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.480391  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.484411 11900 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.486020 12216 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:52.469867 12259 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.486567 12216 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.486951 12259 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.487550 12216 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.478229 11865 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:02:52.489899  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.490696  8576 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:52.491727  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.492386  8576 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:52.492054 12181 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:52.494365  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:52.494877  8624 connection.cc:256] RPC call timeout handler was delayed by -1.4607774649999783s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:02:52.496246  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.496004 12154 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.497210  8728 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:02:52.486233 12244 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:02:52.439664 12224 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 13:02:52.439025 12228 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.454751 12175 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 13:02:52.502511 12258 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.528088  8577 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:52.528016 12258 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.528859  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:52.529506  9323 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.530520  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.532343  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:52.533084 12143 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.533548  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.533834 12158 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:52.512037 12112 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.534281  9330 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:52.534284 12158 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.535328 12137 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.535969  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2414) took 1254 ms. Trace:
I20260709 13:02:52.535588 12158 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.535873 12137 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.536795 12158 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.537593 12112 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.537884 11886 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:52.510449 12220 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:52.499259  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.539882 12111 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:52.540020 12110 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:52.541188 12255 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.462035 12249 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.093s
I20260709 13:02:52.543167 12255 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:52.543720 12255 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.544539  9323 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:52.513343 12245 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.506276  9324 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:52.502110 12216 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.500185 12256 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.518759 11604 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.565817  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:52.566691  9324 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.520296 12224 consensus_queue.cc:1243] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:52.520737  8584 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:52.520934  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 649) took 2528 ms. Trace:
I20260709 13:02:52.517752 12215 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.504477  8696 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:52.502921  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.473567 12237 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.536405  8580 rpcz_store.cc:276] 0709 13:02:51.281537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.281647 (+   110us) service_pool.cc:224] Handling call
0709 13:02:51.281957 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:51.629587 (+347630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.535750 (+906163us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54003e7454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:52.535913 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":782296960}
I20260709 13:02:52.538102 12112 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.538352 11886 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:52.548646  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:52.517683  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.581276  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:52.497382 12154 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.560379  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.584378  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.585677 11929 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:52.586326 11929 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.586810 11929 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.563180  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:52.589500  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:52.589857 11797 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:52.591353 11797 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:52.591981 11797 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.592132 12171 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:52.579108  8582 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:52.593178 12171 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.594817  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.593194 11797 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.550690 11843 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.596714 11797 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.597361 11843 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:52.613603  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.579591  8580 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:52.596776  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.571678  9326 rpcz_store.cc:276] 0709 13:02:49.992604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.992853 (+   249us) service_pool.cc:224] Handling call
0709 13:02:50.001714 (+  8861us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.308436 (+306722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.520582 (+2212146us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400517a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:52.520849 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":648751872}
I20260709 13:02:52.591238  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:52.538364 12137 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:52.676293  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 171868us
W20260709 13:02:52.680781  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.547448 12248 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
W20260709 13:02:52.694232  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 103093us
I20260709 13:02:52.682044 12171 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.694262 12224 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.703981  8580 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:52.705572  8580 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:52.703076  8582 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:52.692705 12237 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:52.710642 12237 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.711107 12237 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.684746 12245 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 13:02:52.715783 12181 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.723719 12228 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.724392  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 663) took 2492 ms. Trace:
W20260709 13:02:52.726811  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2408) took 1452 ms (client timeout 1711 ms). Trace:
W20260709 13:02:52.727231  8575 rpcz_store.cc:269] 0709 13:02:51.278694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.301202 (+ 22508us) service_pool.cc:224] Handling call
0709 13:02:52.726607 (+1425405us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b540002e254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:52.726748 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":595214720}
I20260709 13:02:52.728652  8575 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:52.724773  8733 rpcz_store.cc:276] 0709 13:02:50.232156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.340768 (+108612us) service_pool.cc:224] Handling call
0709 13:02:50.341018 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:51.475342 (+1134324us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:51.475716 (+   374us) log.cc:844] Serialized 14 byte log entry
0709 13:02:52.016796 (+541080us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:52.016866 (+    70us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:52.016940 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.724130 (+707190us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400088ed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:52.724178 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.724306 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":167502,"prepare.queue_time_us":170,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":167632,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"spinlock_wait_cycles":2607303040,"thread_start_us":373008,"threads_started":1,"wal-append.queue_time_us":407125}
I20260709 13:02:52.731655 12215 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:52.731832  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.733897  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:52.675429 11843 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.692584 12261 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:52.738137 12261 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.738675 12261 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:52.739426 12255 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.735780 11929 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.739420 12253 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.741974 12253 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.742746 12140 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.693321  9324 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:52.744699 12253 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:52.570727 12256 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.745927  8575 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:52.747073 12112 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.733889  9326 raft_consensus.cc:2410] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:52.747905 12256 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.706900  8580 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:02:52.748422  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:52.738237 12258 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:52.758682  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 448242us
I20260709 13:02:52.760593  8731 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:52.757710  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.762400  8580 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:52.759833  8583 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:52.759769  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.773999  8656 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:52.807546  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.809473  8583 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:52.810370  8650 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:52.807587  8654 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:52.799476 11929 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.763685 11886 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.830937  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.831658  8658 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:02:52.831828  8732 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:52.835791  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.847043  8650 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:52.846522  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.848173  8651 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:52.848564  8733 raft_consensus.cc:2410] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260709 13:02:52.848980  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 89310.1us
I20260709 13:02:52.849946  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.851987  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:52.852563  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.840994  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2421) took 1529 ms. Trace:
I20260709 13:02:52.855094  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:52.849741  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:52.857833  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:52.853566  8578 rpcz_store.cc:276] 0709 13:02:51.311130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.492165 (+181035us) service_pool.cc:224] Handling call
0709 13:02:52.840785 (+1348620us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54002cb5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:52.840927 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":876159360}
I20260709 13:02:52.858332  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.859735  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:52.859792  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 157765us
I20260709 13:02:52.861443  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.864944  8656 raft_consensus.cc:2410] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:52.867161  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.847721 12218 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:52.871021  8658 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:52.873248  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.853653 11886 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.876971  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.877707  8656 raft_consensus.cc:2376] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8. Re-sending same reply.
I20260709 13:02:52.832760 12265 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.877837  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.861824 12158 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:52.855850 11797 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.863284 12260 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.839551 12259 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.861188 12228 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.882660  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:52.863582 12112 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.863312 11900 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:52.863411 12248 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:52.887282  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.892787  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.835489 12163 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
W20260709 13:02:52.896916 12211 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.620s	user 0.000s	sys 0.006s
I20260709 13:02:52.889775 12158 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.898128 12265 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:52.899585 12248 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:52.900009 12114 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.901378 11900 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.902706  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.872269 12218 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:52.924571 12268 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:52.925472 12268 consensus_queue.cc:1243] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:52.924970 12269 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:52.925735  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.928119  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.879197 12094 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.866859 12092 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
W20260709 13:02:52.909030  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.935204  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.908804 12112 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.918433  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:52.876371 12258 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:02:52.930886  8658 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:52.906899  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.866350 12263 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 13:02:52.942341  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:52.882259 11929 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 13:02:52.934413  8654 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:52.929247  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:52.913615  8651 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
W20260709 13:02:52.917518  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.928885 12114 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.919132 12248 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:52.943943  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.951058  8654 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:52.952986  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.954154  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:52.958847  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:52.961988  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.962672  9326 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:52.965167  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 876) took 2250 ms. Trace:
I20260709 13:02:52.965330  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:52.966414  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:52.950969 12228 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.967798 12228 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:52.968405 12228 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.969547 12228 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.970813 12158 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.973014 12092 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:52.973322  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:52.975037 12092 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:52.975432 12092 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.975754 12092 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:52.976043  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:52.977138 11900 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.982295 12261 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.956573 12112 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:52.985471 12261 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.950534 11604 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:52.986975 11604 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:52.987942 11558 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:52.988091 11604 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.988811  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.951176 12140 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.950876 12094 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:52.991209 12094 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:52.992359 11891 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:52.994138 12119 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:52.994386 12094 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:52.994630 12119 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:52.951262 12265 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:52.995514 12265 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:52.995602 12119 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:52.997859  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:52.965618  9331 rpcz_store.cc:276] 0709 13:02:50.714666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.714812 (+   146us) service_pool.cc:224] Handling call
0709 13:02:50.766793 (+ 51981us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:50.767328 (+   535us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.767372 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:50.795887 (+ 28515us) log.cc:844] Serialized 52 byte log entry
0709 13:02:51.052064 (+256177us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:51.748755 (+696691us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b1800070b40. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf
0709 13:02:51.749010 (+   255us) log.cc:844] Serialized 14 byte log entry
0709 13:02:51.749306 (+   296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.795784 (+ 46478us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:52.862140 (+1066356us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:02:52.964933 (+102793us) raft_consensus.cc:1685] finished
0709 13:02:52.964957 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.965110 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":225047,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":4345,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":102115,"spinlock_wait_cycles":37252224,"thread_start_us":369715,"threads_started":2,"wal-append.queue_time_us":382698}
I20260709 13:02:53.000612  9324 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:53.001264  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:53.002224  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43496.6us
W20260709 13:02:53.003662  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.003827  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:53.011009  8654 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:02:53.011756  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.013121  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:53.016242  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.016409  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:53.017436  8726 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:53.018527  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:53.019347  8729 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:53.023095  8728 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:52.996601 12265 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.025835  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:53.026932  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:52.997234 12119 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:53.019124  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:53.039225  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:53.040005  8576 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:53.028481  8730 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:53.030514  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:53.031512  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:53.035522  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:53.036899  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:53.047092  8583 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:53.048014  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.048511  8583 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:53.049291  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.050616  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.051234  8583 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:02:53.009737 11604 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:53.045213  8578 raft_consensus.cc:2376] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6. Re-sending same reply.
W20260709 13:02:53.054864  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 89593.8us
I20260709 13:02:53.035571 11704 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:53.058110 11704 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:53.058739  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:53.059162 11569 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:53.060213  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:53.060379 11704 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:53.060784 11704 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:53.061318 11569 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:53.057253  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.061570 12269 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:53.028213 12154 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:53.063082  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:53.058815 12276 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 13:02:53.034834 12115 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:53.067274  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2487) took 2083 ms. Trace:
I20260709 13:02:53.069655  8577 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:53.067903  8657 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:53.071813  8655 rpcz_store.cc:276] 0709 13:02:50.983456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.012684 (+ 29228us) service_pool.cc:224] Handling call
0709 13:02:51.012989 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:51.353780 (+340791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.066936 (+1713156us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b5400001754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:53.067168 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":709760896}
I20260709 13:02:53.070005  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:53.074486  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.070714  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.074728  8651 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:53.073316  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:53.075199  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:53.076279  8656 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:02:53.074126  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.079162  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.025161 12237 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.022658 12264 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:02:53.082623  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:53.074400 12270 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 13:02:53.066998 11569 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:53.028401 12219 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 13:02:53.087776  8577 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 13:02:53.090435  8696 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.091421 12224 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:53.092510 12264 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:53.093856 11558 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:53.085873  8578 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:53.062547 12163 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:53.075904 12115 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.075706 12110 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:53.102636 11569 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:53.066388 12269 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:53.106005  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:53.086418 12128 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:53.113778  8576 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:53.126236  8659 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:53.132124  8576 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:53.082521 12154 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:53.100658  8580 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260709 13:02:53.117446  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.104167 12224 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:53.134622 12154 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:53.135106 12224 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:53.135157 12154 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:53.135669  8580 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:53.136837  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.111093 11558 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:53.121484 12272 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
I20260709 13:02:53.117749 12115 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:53.078749 11704 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.128214 12111 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:53.118039 12110 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.152038 12293 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:53.115367 12181 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20260709 13:02:53.154767  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.136736 12128 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.137029 12158 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:53.150437 12114 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:53.150933 12264 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:53.158339 12110 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:53.167054 12128 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:53.185073  8729 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 5.
I20260709 13:02:53.192951  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 684) took 1053 ms. Trace:
I20260709 13:02:53.193413  8582 rpcz_store.cc:276] 0709 13:02:52.139459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.707848 (+568389us) service_pool.cc:224] Handling call
0709 13:02:52.708359 (+   511us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.709024 (+   665us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:52.709059 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:52.709608 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 13:02:52.804177 (+ 94569us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:52.804218 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.804288 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:53.192707 (+388419us) raft_consensus.cc:1685] finished
0709 13:02:53.192726 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.192888 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":89,"thread_start_us":94147,"threads_started":1,"wal-append.queue_time_us":146827}
I20260709 13:02:53.199258 12297 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:53.199869 12297 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:53.201081 12284 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:02:53.202153 12297 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:53.203117 12275 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 13:02:53.205643 11558 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:53.208252 12210 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.908s	user 0.004s	sys 0.000s
I20260709 13:02:53.214452 12237 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:53.216044 12225 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.358s	user 0.006s	sys 0.001s
I20260709 13:02:53.219349 12248 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.221278  8731 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:53.176378 11704 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:53.226159 12171 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.521s	user 0.012s	sys 0.000s
W20260709 13:02:53.225463 12273 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 13:02:53.272728 12228 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.273578 12228 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.276206 12228 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:53.263902 12171 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:53.271716 12294 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:53.175859 12269 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.288924 12163 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:53.307502 12293 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:53.324075 12293 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.315480 12294 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.336625 12209 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.338549 12209 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:53.341329  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 679) took 1234 ms. Trace:
I20260709 13:02:53.342465  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 687) took 1185 ms. Trace:
I20260709 13:02:53.411782  8575 rpcz_store.cc:276] 0709 13:02:52.162694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.732462 (+569768us) service_pool.cc:224] Handling call
0709 13:02:52.732764 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:52.746384 (+ 13620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.342118 (+595734us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54000f96d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:53.342387 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29817728}
I20260709 13:02:53.348050  8725 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:53.413753  8725 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:53.357837  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 332350us
W20260709 13:02:53.359580  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.394930  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2465) took 1281 ms. Trace:
W20260709 13:02:53.400722  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 518004us
I20260709 13:02:53.420730  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:53.421824  8733 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:53.424252  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:53.425197  8651 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:53.426620  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:53.427379  8727 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:53.409020  9294 leader_election.cc:400] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6.
I20260709 13:02:53.430936  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:53.431387  9294 leader_election.cc:403] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 13:02:53.435019  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:53.418860  8584 rpcz_store.cc:276] 0709 13:02:52.113298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.278169 (+164871us) service_pool.cc:224] Handling call
0709 13:02:53.394667 (+1116498us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54000b6154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:53.394846 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":468624256}
I20260709 13:02:53.410260  8579 rpcz_store.cc:276] 0709 13:02:52.106721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.313502 (+206781us) service_pool.cc:224] Handling call
0709 13:02:53.341083 (+1027581us) spinlock_profiling.cc:243] Waited 8.24 ms on lock 0x7b540030c354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:53.341245 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18954624}
I20260709 13:02:53.448230  8733 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:53.416884  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:02:53.456355  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47590us
I20260709 13:02:53.457194  9294 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.357488 12103 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.480378  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:53.481436  9331 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:02:53.481897  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.482230  8584 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 13:02:53.482751  8584 pending_rounds.cc:85] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 6
I20260709 13:02:53.483031  8584 pending_rounds.cc:107] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:02:53.483371  8584 raft_consensus.cc:2889] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:53.454187  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:53.485105  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:53.414805  8725 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.464773  8652 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:02:53.480249  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:53.460953  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:53.498335  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:53.499817  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:53.500735  8651 raft_consensus.cc:2410] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:53.501423  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:53.486865  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:53.502439  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:53.503006  8575 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:53.503039  8732 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:53.505980  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:53.507134  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:53.512406  8727 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:53.356926 12258 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:53.356501 12013 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:53.435606 12237 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.505546  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:53.517992  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:53.520728  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:53.385361 12248 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:53.527488  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:53.527765  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:53.523422  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.514730  8725 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:53.528365  8659 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:53.530865  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:53.532243  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:53.480731 12253 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:53.461066 12130 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6.
I20260709 13:02:53.534286  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:53.536761  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.514253 12290 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.518133 12261 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:53.534742 11891 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.542s	user 0.013s	sys 0.012s
I20260709 13:02:53.544886  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:53.613494 12248 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:53.618327  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 73527us
I20260709 13:02:53.618698 12248 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.620069  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:53.548691  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:53.547094  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2365) took 1352 ms. Trace:
I20260709 13:02:53.627600  8657 rpcz_store.cc:276] 0709 13:02:52.195112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.204518 (+  9406us) service_pool.cc:224] Handling call
0709 13:02:53.067481 (+862963us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.072723 (+  5242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.546871 (+474148us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b5400001750. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:53.547026 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1940939008}
I20260709 13:02:53.550786  8725 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:53.629019  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2376) took 1135 ms. Trace:
I20260709 13:02:53.501185 12103 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.556362  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:53.630376  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 74082.1us
I20260709 13:02:53.629339  8725 rpcz_store.cc:276] 0709 13:02:52.494104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.502653 (+  8549us) service_pool.cc:224] Handling call
0709 13:02:52.516080 (+ 13427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.347899 (+831819us) spinlock_profiling.cc:243] Waited 55.2 ms on lock 0x7b5400156920. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:02:53.628741 (+280842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.628955 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":126847360}
I20260709 13:02:53.591501  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:53.547993 12219 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.555794  8732 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:53.535198 12292 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:53.518724 12258 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:53.588464 12092 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.602s	user 0.012s	sys 0.000s
W20260709 13:02:53.597154  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.613768 12261 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.621843  9330 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:53.395720 12171 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.128s
I20260709 13:02:53.669951 12171 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:53.670680 12171 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:53.671185 12171 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.622612 12130 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:53.623425 11704 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:53.604352  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.681739  8727 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:53.547847 12253 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.534929 12237 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.547075 12140 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 13:02:53.625068  8729 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:53.586977 11900 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:53.625756  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 885) took 1123 ms. Trace:
I20260709 13:02:53.480594 12259 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 13:02:53.636363 11604 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:53.546645 12268 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:53.621152 12248 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:53.666270  8725 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:02:53.615172 12099 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:53.638278 12216 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:53.675921  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 127254us
I20260709 13:02:53.556061 11569 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.678010  8731 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:53.526732 12163 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.638430 12219 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.677572 12171 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:53.685920 12303 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 13:02:53.677680 12112 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 13:02:53.688154  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:53.582794 12290 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:53.675105 12119 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.621s	user 0.009s	sys 0.004s
I20260709 13:02:53.719471 12130 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:53.707909  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 87873.4us
W20260709 13:02:53.708312  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 203963us
I20260709 13:02:53.620882 12224 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.665740 12265 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:53.674355 12261 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:53.663877 12292 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:53.717074  8727 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:53.662582 12094 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.662848 11891 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.719142  8725 pending_rounds.cc:85] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 3
I20260709 13:02:53.746500  8725 pending_rounds.cc:107] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:02:53.746830  8725 raft_consensus.cc:2889] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:53.744309  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:53.722999  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:53.681644 11704 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:53.686564 12253 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:53.637240 12228 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:53.666523 12092 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:53.689913  9327 rpcz_store.cc:276] 0709 13:02:52.502188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.502368 (+   180us) service_pool.cc:224] Handling call
0709 13:02:52.502665 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.000010 (+497345us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:53.032176 (+ 32166us) log.cc:844] Serialized 14 byte log entry
0709 13:02:53.358361 (+326185us) log.cc:844] Serialized 14 byte log entry
0709 13:02:53.358628 (+   267us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:53.358692 (+    64us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:53.358765 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.625504 (+266739us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b5400693a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:53.625551 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.625698 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":316126,"prepare.queue_time_us":33939,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":320064,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":143,"spinlock_wait_cycles":1149608576,"thread_start_us":9299,"threads_started":1,"wal-append.queue_time_us":36668}
I20260709 13:02:53.732371 12281 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 13:02:53.739287  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:53.998944  8726 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:53.721043  9329 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:53.769168  8730 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:53.764443 11891 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:53.765339 12094 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:53.817814  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:53.860363  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:53.917959  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.820122 12293 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.987587 12099 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:53.990487  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 510433us
I20260709 13:02:53.734494 12316 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:53.725312 12268 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:54.002724  8728 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:54.003285  8731 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:54.002171 11569 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.014902 12292 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:54.015764 12299 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.401s	user 0.001s	sys 0.000s
I20260709 13:02:54.016834 12294 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.017836 12312 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:54.057646  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 314995us
I20260709 13:02:54.177961 12312 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:53.978091 12100 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 13:02:54.179723  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:54.180502  9325 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:54.180102 12312 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.180902  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:54.123195  8578 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53"
dest_uuid: "a0d703ee18464475a290425976246555"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:60002
W20260709 13:02:54.069694  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.028s	sys 0.054s
I20260709 13:02:54.168347 12099 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.182387  8578 raft_consensus.cc:606] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:54.182842  8578 raft_consensus.cc:614] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:02:54.120774  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.185161  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.185484 12103 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:53.977487 11704 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:54.186934 12315 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 13:02:54.079684  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.122641  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:54.179314  8726 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:54.004657 12245 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.166s
I20260709 13:02:53.731804 12258 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:54.114128 12154 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.964s	user 0.010s	sys 0.002s
W20260709 13:02:54.164791  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 441747us
I20260709 13:02:54.201213  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.066234 12127 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.067113 12228 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:54.202068 12228 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:54.202534 12228 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:54.203493 12099 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.065168 12112 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:54.204108 11891 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:54.204277 12112 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.173005 11569 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.206139 11569 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.206640 12112 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.170681 12130 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.171258 12131 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:02:54.182462  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.038s	sys 0.060s
W20260709 13:02:54.209131  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 676816us
W20260709 13:02:54.121224  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51565.9us
I20260709 13:02:54.114398 12293 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.188050  9325 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:54.190822  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:54.215543  9325 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:02:54.160563  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 137108us
I20260709 13:02:54.169478 12316 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:02:54.191793  8622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.000s	sys 0.021s
W20260709 13:02:54.220177  8622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.000s	sys 0.021s
I20260709 13:02:54.004884 12259 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.163s
I20260709 13:02:54.200889 12154 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.201459 12258 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.206230 12268 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:54.179807 12313 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 13:02:54.233903  8732 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260709 13:02:54.235379  8658 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 13:02:54.174873 12115 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.016s	user 0.011s	sys 0.000s
I20260709 13:02:54.214773  8727 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:54.224764  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:54.196352 12103 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:54.180505 12237 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.235504  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:54.236366  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:54.236469  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:54.274051  8650 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:02:54.223338 12128 consensus_meta.cc:287] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.999s	user 0.005s	sys 0.005s
I20260709 13:02:54.275102  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:54.276502  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:54.278236  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:54.278843  8657 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:54.237203  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:54.179317 12290 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.241760  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.210142 12265 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.223526 12316 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.247684  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:54.246758 12219 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.248720 12293 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.208395 11569 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.249797 12154 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:54.249711  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:54.294960  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:54.249665 12115 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:54.251884 12314 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 13:02:54.254151  8724 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:54.254894  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:54.261286  9323 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260709 13:02:54.240219 12127 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:54.262290  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.267261 12259 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:54.269258  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2478) took 1247 ms. Trace:
I20260709 13:02:54.196475 12092 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:54.272640  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:54.282186  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.225147 12112 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:54.288884  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 263260us
W20260709 13:02:54.344340  8625 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.751s (SENT) [suppressed 28 similar messages]
W20260709 13:02:54.346134  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:54.273577 12013 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.017s	sys 0.021s
W20260709 13:02:54.287551 12110 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.093s	user 0.012s	sys 0.010s
I20260709 13:02:54.300112  8726 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:54.335259  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.341730  8658 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:54.278080 12237 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:54.332756  8576 rpcz_store.cc:276] 0709 13:02:53.022080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.133434 (+111354us) service_pool.cc:224] Handling call
0709 13:02:53.133929 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:53.136648 (+  2719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:53.136674 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:53.150600 (+ 13926us) log.cc:844] Serialized 26 byte log entry
0709 13:02:53.307919 (+157319us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:53.307961 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.308053 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.268993 (+960940us) raft_consensus.cc:1685] finished
0709 13:02:54.269016 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.269182 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2151,"raft.queue_time_us":2257,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":13744,"thread_start_us":121141,"threads_started":1,"wal-append.queue_time_us":342989}
I20260709 13:02:54.348879  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:54.281896 11891 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:02:54.354560  9323 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:54.358340  8655 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260709 13:02:54.360569  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 78456.4us
I20260709 13:02:54.283696 12128 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.367736  8658 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:02:54.368839  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.331645 12317 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:54.377306  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:54.377628 12092 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:54.380714 12092 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:54.380702 12322 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:54.349851 12013 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.017s	sys 0.022s
I20260709 13:02:54.342784 12140 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.346136 12259 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:54.394905 12320 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:54.293036 12265 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.353981 12318 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:54.408797  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 193984us
I20260709 13:02:54.355571 12245 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:54.381256 12092 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:54.279532 12103 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.418846  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:54.386998 12219 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:02:54.423410  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 238298us
I20260709 13:02:54.384900 12228 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.357916 12110 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.443027  8658 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:54.345441 12319 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 13:02:54.445042  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.446566  8658 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:02:54.447395  8650 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:54.353892 12127 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.449014  8657 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:54.421042 12092 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.452241  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:54.395745 12140 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.414511 12100 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 13:02:54.446859 11569 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:54.471194  8659 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:02:54.423908 12294 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:02:54.477358  9327 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:54.449577 12137 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 13:02:54.440574 12312 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:02:54.440778 12119 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 13:02:54.455838 12245 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:54.466639 12320 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:54.487213  9323 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "04399390cf414886a3453981dc1f74aa"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:02:54.493575  9323 raft_consensus.cc:606] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:54.494087  9323 raft_consensus.cc:614] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:54.473932 11569 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:54.422367 12318 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.389452 12013 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:54.495724 12320 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.515478 12319 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:54.515882 12115 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.516371 12255 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:02:54.532614  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 162519us
W20260709 13:02:54.519858  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 140467us
I20260709 13:02:54.500941 11569 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:54.482270 12258 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 13:02:54.542129  8576 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:02:54.503211 12321 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:54.536273  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:54.521199 12137 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:54.567194 12137 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:54.572427 12265 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.572095 12245 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.558645 12258 consensus_queue.cc:1243] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:54.573225 12137 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.567924 12332 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.574867 12321 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:54.574493 12332 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.575641 12128 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.575453 12321 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.544859 12319 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.578366 12319 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.578933 12319 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.580334 12319 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.559264 12255 consensus_queue.cc:1243] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:54.570981 12324 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 13:02:54.582831 12324 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:54.583263 12324 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:54.584337 12324 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.574622 12258 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:54.485605 12259 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.594662  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.595597  8576 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:54.597951  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.598114  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.598656  8575 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:54.588269 12137 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:54.588894 12209 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.250s	user 0.008s	sys 0.006s
I20260709 13:02:54.606684  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.552981 11704 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.607568  8576 raft_consensus.cc:2410] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 11 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:54.612658  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:54.613430  8576 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:54.588124 12140 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.619753  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.610656  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.609773  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:54.621347  8575 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:02:54.621418  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 137892us
W20260709 13:02:54.611490  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 139972us
I20260709 13:02:54.617588  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:54.622713  8650 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:54.595479 12265 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.619650  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:54.629577  8657 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:02:54.628486 12103 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.632380 12320 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.640998  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:54.654639  8580 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.607563 12324 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.659469  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.660166  8582 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:54.581506 12334 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 13:02:54.660601  8576 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:54.680310  8577 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:54.681635  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:54.687491  8650 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:02:54.628296 12110 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.628842 12228 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.638767 12112 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.660436 12323 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.717350  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:54.721298  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 185056us
I20260709 13:02:54.683149 12127 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.735271  8582 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 5: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:54.640300 12265 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.661074 12321 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.734598  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:54.738780  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:54.727427 12352 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.620725 12013 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.725992 12342 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.725265 12320 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.731931 12323 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.682330 12119 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 13:02:54.764499  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:54.737051 12228 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:54.775964 12261 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.008s	user 0.010s	sys 0.004s
I20260709 13:02:54.771895  8581 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:54.791296  8580 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:02:56.259205  8580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2486) took 1643 ms (client timeout 1706 ms). Trace:
W20260709 13:02:56.259675  8580 rpcz_store.cc:269] 0709 13:02:54.615307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.625544 (+ 10237us) service_pool.cc:224] Handling call
0709 13:02:56.259119 (+1633575us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.260865  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:54.918078 12342 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:54.773316 12352 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:55.459131 12360 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:56.541863 12114 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:56.541967 12374 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.542413 12114 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:56.542678 12374 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.543339 12114 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.544624 12114 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:56.544955 12374 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:55.463352 12363 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.549480 12363 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.551837 12363 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.552345 12363 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.552745 12363 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:56.554965  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.79015e+06us
I20260709 13:02:56.557035 12154 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.557890 12132 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:56.557813 12154 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.558470 12132 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.560436 12132 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.560842 12293 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:56.561479 12293 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:56.562454 12293 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:55.449340  8471 maintenance_manager.cc:419] P b2a6e2bdaa2043fba12c0ef60545b7fb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033411
W20260709 13:02:55.200106  8479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 305) took 23965 ms (client timeout 15000 ms). Trace:
I20260709 13:02:55.464185  8582 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:55.417330  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:55.384985  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.580219 12171 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:55.462984  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2505) took 1963 ms. Trace:
I20260709 13:02:55.409894  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2372) took 2377 ms. Trace:
I20260709 13:02:56.583216  8656 rpcz_store.cc:276] 0709 13:02:53.032659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.084328 (+ 51669us) service_pool.cc:224] Handling call
0709 13:02:53.084581 (+   253us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:53.144020 (+ 59439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:53.144051 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:53.154679 (+ 10628us) log.cc:844] Serialized 52 byte log entry
0709 13:02:53.227259 (+ 72580us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:53.265158 (+ 37899us) log.cc:844] Serialized 14 byte log entry
0709 13:02:53.271291 (+  6133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.271431 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:55.105982 (+1834551us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b18003da680. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b29522 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:55.112089 (+  6107us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b18003da680. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:02:55.409539 (+297450us) raft_consensus.cc:1685] finished
0709 13:02:55.409563 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.409798 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5802,"prepare.queue_time_us":40051,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":55,"raft.queue_time_us":45584,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":17074,"spinlock_wait_cycles":15420416,"thread_start_us":106971,"threads_started":2,"wal-append.queue_time_us":134968,"wal-append.run_cpu_time_us":10693,"wal-append.run_wall_time_us":1003931}
I20260709 13:02:55.662185  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1911) took 1064 ms. Trace:
I20260709 13:02:56.197157  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:56.586191  8581 rpcz_store.cc:276] 0709 13:02:54.597750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.597887 (+   137us) service_pool.cc:224] Handling call
0709 13:02:55.661998 (+1064111us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54002ca1d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:55.662105 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":398189312}
I20260709 13:02:56.587001  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 697) took 1885 ms. Trace:
I20260709 13:02:56.587414  8578 rpcz_store.cc:276] 0709 13:02:54.701793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.196860 (+1495067us) service_pool.cc:224] Handling call
0709 13:02:56.586940 (+390080us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:56.236135  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.502s	user 0.016s	sys 0.036s
I20260709 13:02:56.588250  8650 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:02:56.588689  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.854s	user 0.016s	sys 0.037s
W20260709 13:02:55.833981  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.063s	user 0.019s	sys 0.018s
W20260709 13:02:56.591269  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.820s	user 0.021s	sys 0.018s
I20260709 13:02:56.593454 12360 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:56.593341  8550 connection.cc:256] RPC call timeout handler was delayed by -1.0159990749999679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260709 13:02:56.594286 12360 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:55.985347  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 81369.1us
I20260709 13:02:55.536553  9328 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 13:02:55.464385  9329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 880) took 2010 ms (client timeout 1691 ms). Trace:
W20260709 13:02:56.606231  9329 rpcz_store.cc:269] 0709 13:02:53.454280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.460708 (+  6428us) service_pool.cc:224] Handling call
0709 13:02:55.464182 (+2003474us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400518454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:55.464286 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516016000}
I20260709 13:02:56.607581  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:55.371868  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 890) took 2412 ms. Trace:
W20260709 13:02:56.609241  8698 connection.cc:256] RPC call timeout handler was delayed by -2.0986781839999935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:02:56.609050  9324 rpcz_store.cc:276] 0709 13:02:52.959122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.959372 (+   250us) service_pool.cc:224] Handling call
0709 13:02:52.959684 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.001156 (+ 41472us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.371522 (+2370366us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b54004dec54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:55.371763 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93591936}
I20260709 13:02:56.610471  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:56.611004  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 899) took 1931 ms (client timeout 2249 ms). Trace:
W20260709 13:02:56.611244  9324 rpcz_store.cc:269] 0709 13:02:54.679558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.610272 (+1930714us) service_pool.cc:224] Handling call
0709 13:02:56.610957 (+   685us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.611800  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.612174  9324 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:56.612787  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:55.506541  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 879) took 2053 ms. Trace:
I20260709 13:02:56.613883  9330 rpcz_store.cc:276] 0709 13:02:53.453145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.516996 (+ 63851us) service_pool.cc:224] Handling call
0709 13:02:55.506330 (+1989334us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b5400386b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:55.506467 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":169898240}
I20260709 13:02:56.615020  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.615720 12114 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:56.220399  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.620630  9323 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:56.621124  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 883) took 1963 ms (client timeout 1970 ms). Trace:
W20260709 13:02:56.186610  9352 debug-util.cc:398] Leaking SignalData structure 0x7b0800296360 after lost signal to thread 7481
I20260709 13:02:56.242194 11558 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:54.989264 11569 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.998431 11704 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:55.475958 11797 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:02:56.638513  8698 connection.cc:256] RPC call timeout handler was delayed by -1.0797311959999547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
I20260709 13:02:56.638536 11704 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:56.639358 11558 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:56.640300 12362 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.640023 11558 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.641013 12362 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.643102 11558 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.644090 11558 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:55.183408 11843 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:02:55.030864 12092 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.000s
I20260709 13:02:55.575532 12094 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.728148 12112 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:55.075410 12130 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.001s	sys 0.001s
I20260709 13:02:55.040995 12131 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:56.655721 12131 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:54.767987 12163 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:56.179597 12115 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:56.197111 12156 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:54.929013 12228 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.667290 12228 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.660030 12115 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:55.024121 12253 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.020s	user 0.010s	sys 0.000s
I20260709 13:02:56.674614 12253 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:55.015069 12256 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.940s	user 0.021s	sys 0.021s
W20260709 13:02:56.678987 12256 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.607s	user 0.022s	sys 0.022s
I20260709 13:02:56.679824  9325 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:02:55.522677 12260 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:54.933143 12261 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:55.028584 12265 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:55.405525 12290 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.688966 12256 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:55.167510 12311 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.003s	sys 0.000s
W20260709 13:02:55.165581 12317 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.003s	sys 0.002s
I20260709 13:02:54.990329 12318 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:54.921312 12320 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:55.035625 12322 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.653s	user 0.008s	sys 0.004s
I20260709 13:02:55.026738 12323 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:54.983609 12316 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.684s	user 0.011s	sys 0.000s
I20260709 13:02:54.922117 12319 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:54.928726 12334 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:55.017439 12338 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:56.704717 12316 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.711184 12338 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:54.979610 12339 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.018080 12340 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.718719 12340 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:55.501921 12343 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:55.018760 12350 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:56.725033 12350 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:55.016849 12351 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
W20260709 13:02:54.981853 12344 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 13:02:54.770469 12345 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:56.734026 12345 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:56.734596 12345 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.980276 12349 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.262359  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2482) took 1651 ms. Trace:
I20260709 13:02:56.271883 11955 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:56.274119 12376 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.278126 12268 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.292156  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 1914) took 1683 ms. Trace:
I20260709 13:02:56.303695  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2509) took 2068 ms. Trace:
W20260709 13:02:56.323846  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2369) took 3357 ms (client timeout 1623 ms). Trace:
W20260709 13:02:56.327834  8548 connection.cc:256] RPC call timeout handler was delayed by -1.4114337490000253s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
I20260709 13:02:56.330515 12335 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.339205  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2061) took 2611 ms. Trace:
I20260709 13:02:56.352782  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:56.363803  8576 raft_consensus.cc:2376] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7. Re-sending same reply.
I20260709 13:02:56.366366  8584 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:56.746052  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2487) took 2130 ms. Trace:
I20260709 13:02:56.737629  8580 rpcz_store.cc:276] 0709 13:02:54.610620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.260592 (+1649972us) service_pool.cc:224] Handling call
0709 13:02:56.262297 (+  1705us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.737816 11955 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:56.747834 11955 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:56.412866  8481 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 5 to 7, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 7 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:56.750555  8481 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:56.438781 12355 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.443s	user 0.000s	sys 0.013s
I20260709 13:02:56.443190 12269 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:56.452267 11891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.048s	user 0.010s	sys 0.018s
W20260709 13:02:56.754971 11891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.365s	user 0.010s	sys 0.019s
I20260709 13:02:56.464457 12224 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:56.757130 12175 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:56.475536  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:56.758198  9322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 897) took 2276 ms (client timeout 2313 ms). Trace:
I20260709 13:02:56.742040  8577 rpcz_store.cc:276] 0709 13:02:54.609012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.609870 (+   858us) service_pool.cc:224] Handling call
0709 13:02:54.680142 (+ 70272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.680782 (+   640us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.291901 (+1611119us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b54000f91d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:56.292077 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92454784}
W20260709 13:02:56.758656  9322 rpcz_store.cc:269] 0709 13:02:54.482253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.475242 (+1992989us) service_pool.cc:224] Handling call
0709 13:02:56.758120 (+282878us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.759622  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:56.760216  9322 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:56.486289 12353 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.762113 12353 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.764407 12353 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.744657 12323 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.753233  8481 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 4 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:56.765806 12323 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.744928  8730 rpcz_store.cc:276] 0709 13:02:53.727441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.727579 (+   138us) service_pool.cc:224] Handling call
0709 13:02:53.727887 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.003404 (+275517us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.338887 (+2335483us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b5400089dd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:56.339117 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92254592}
I20260709 13:02:56.737102 12350 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.767942  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:56.768482 12323 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:55.342655  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:56.771625 12382 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.772145 12382 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.520220 12312 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:56.774550 12312 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:56.775403 12312 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.776417 12312 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:56.531690 12255 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.532222 12342 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.777686 12255 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:56.532414  8625 connection.cc:256] RPC call timeout handler was delayed by -1.3586335249999593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
I20260709 13:02:56.779433 12342 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.779986 12342 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.538492 12352 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.561069 12132 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.781793 12132 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.782583 12342 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.783195 12332 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.742213  8659 rpcz_store.cc:276] 0709 13:02:54.235206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.236361 (+  1155us) service_pool.cc:224] Handling call
0709 13:02:54.246319 (+  9958us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:54.475047 (+228728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.303389 (+1828342us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540018b0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:56.303602 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510649088}
I20260709 13:02:56.784895 12132 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:56.577098  8582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 698) took 1855 ms (client timeout 2013 ms). Trace:
I20260709 13:02:56.757616 12175 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:02:56.787066  8582 rpcz_store.cc:269] 0709 13:02:54.719521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.738410 (+ 18889us) service_pool.cc:224] Handling call
0709 13:02:56.574727 (+1836317us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b540002e754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:56.574862 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1615369088}
I20260709 13:02:56.571451 12363 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.580928 12171 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.787842 12175 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.582387  8655 rpcz_store.cc:276] 0709 13:02:53.500023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.504769 (+  4746us) service_pool.cc:224] Handling call
0709 13:02:53.505106 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:54.367370 (+862264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.462618 (+1095248us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b54000591d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:55.462894 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1961263872}
I20260709 13:02:56.590080  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:55.441211  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2056) took 1891 ms. Trace:
I20260709 13:02:56.187605  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2049) took 2768 ms. Trace:
I20260709 13:02:56.594942 11900 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:56.597164 12360 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.792732  8729 rpcz_store.cc:276] 0709 13:02:53.419060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.419214 (+   154us) service_pool.cc:224] Handling call
0709 13:02:53.624873 (+205659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.689431 (+ 64558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.187346 (+2497915us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b5400436f50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:56.187529 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471865728}
I20260709 13:02:56.792979 11900 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.794260  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:56.794991  8729 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:56.795660 11900 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.601830 12154 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:56.571732  8479 rpcz_store.cc:269] 0709 13:02:31.235105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.241039 (+  5934us) service_pool.cc:224] Handling call
0709 13:02:55.200007 (+23958968us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:48.013784 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:48.014495 (+   711us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:48.014545 (+    50us) write_op.cc:435] Acquired schema lock
0709 13:02:48.014564 (+    19us) tablet.cc:662] Decoding operations
0709 13:02:48.038380 (+ 23816us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:48.038484 (+   104us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:48.038518 (+    34us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:02:48.039880 (+  1362us) tablet.cc:701] Row locks acquired
0709 13:02:48.039928 (+    48us) write_op.cc:260] PREPARE: finished
0709 13:02:48.040112 (+   184us) write_op.cc:270] Start()
0709 13:02:48.040266 (+   154us) write_op.cc:276] Timestamp: P: 1783602168040045 usec, L: 0
0709 13:02:48.040288 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:02:48.041096 (+   808us) log.cc:844] Serialized 5457 byte log entry
0709 13:02:49.067254 (+1026158us) op_driver.cc:464] REPLICATION: finished
0709 13:02:49.364630 (+297376us) write_op.cc:301] APPLY: starting
0709 13:02:49.411470 (+ 46840us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:49.553095 (+141625us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:49.605225 (+ 52130us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:49.959641 (+354416us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:49.960011 (+   370us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0709 13:02:49.960070 (+    59us) write_op.cc:312] APPLY: finished
0709 13:02:49.960459 (+   389us) log.cc:844] Serialized 101 byte log entry
0709 13:02:50.037647 (+ 77188us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:50.076622 (+ 38975us) write_op.cc:454] Released schema lock
0709 13:02:50.122708 (+ 46086us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:50.165605 (+ 42897us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:50.173508 (+  7903us) write_op.cc:454] Released schema lock
0709 13:02:51.318671 (+1145163us) spinlock_profiling.cc:243] Waited 75.9 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
Metrics: {"mutex_wait_us":7488829,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":254632,"apply.run_cpu_time_us":28957,"apply.run_wall_time_us":790452,"num_ops":14,"prepare.queue_time_us":280219,"prepare.run_cpu_time_us":40684,"prepare.run_wall_time_us":474969,"raft.queue_time_us":385695,"raft.run_cpu_time_us":35842,"raft.run_wall_time_us":150991,"replication_time_us":1026751,"spinlock_wait_cycles":174550400,"thread_start_us":971188,"threads_started":5,"wal-append.queue_time_us":679461,"wal-append.run_cpu_time_us":16225,"wal-append.run_wall_time_us":3640224}]]}
W20260709 13:02:56.603360  8697 connection.cc:256] RPC call timeout handler was delayed by -2.1525496179999664s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
I20260709 13:02:56.796396 12335 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:56.797606  8697 connection.cc:256] RPC call timeout handler was delayed by -1.3700502309999365s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
W20260709 13:02:56.797833  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 445060us
I20260709 13:02:56.799226 12154 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.799742 12154 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.616163  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:56.617616  9329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 900) took 1896 ms (client timeout 2087 ms). Trace:
I20260709 13:02:56.801524  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.801684 12363 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.440687  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:56.802203 12363 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.803435 11704 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:02:56.803797  9331 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 884) took 2143 ms (client timeout 1783 ms). Trace:
W20260709 13:02:56.804168  9331 rpcz_store.cc:269] 0709 13:02:54.659976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.440414 (+780438us) service_pool.cc:224] Handling call
0709 13:02:56.803718 (+1363304us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.805015 12363 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.624415  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:56.624856 12158 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.815150 12158 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.625154  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:56.625367  9323 rpcz_store.cc:269] 0709 13:02:54.657498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.220057 (+1562559us) service_pool.cc:224] Handling call
0709 13:02:56.621065 (+401008us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.816744  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.816821  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:56.817425  9323 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:56.817693 12363 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:56.818228 12363 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.818509  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.625702  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:56.819991  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:56.626022  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:56.628711 12358 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:56.820423 12343 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:56.629154 12304 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.629869 12383 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.637841  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:56.646744 11843 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.647261 12114 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.647933 12092 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.637972 11569 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.645268 12362 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:56.654604 12130 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.265s	user 0.003s	sys 0.001s
I20260709 13:02:56.652172 12112 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.640337 12384 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.670291 12115 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.657714 12163 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.687374 12261 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.689447 12256 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:56.690073 12253 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.689380 12290 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.877436 12256 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.879140 12256 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:56.699105 12320 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:56.701258 12311 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.475s	user 0.006s	sys 0.000s
I20260709 13:02:56.698951 12131 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.703078 12322 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:56.701107 12317 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.223s	user 0.004s	sys 0.004s
I20260709 13:02:56.710124 12334 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.713217 12316 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.746641  8584 rpcz_store.cc:276] 0709 13:02:54.615601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.365674 (+1750073us) service_pool.cc:224] Handling call
0709 13:02:56.366045 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.745671 (+379626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.745969 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.377846  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2382) took 3351 ms. Trace:
I20260709 13:02:56.905030  8728 rpcz_store.cc:276] 0709 13:02:53.026796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.026991 (+   195us) service_pool.cc:224] Handling call
0709 13:02:53.027327 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.166417 (+1139090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.377446 (+2211029us) spinlock_profiling.cc:243] Waited 975 ms on lock 0x7b54002049d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:56.377752 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2242375040}
I20260709 13:02:56.907122  8728 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260709 13:02:56.744570 12376 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.488894 12341 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.744660 12268 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.491886 12218 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.769675  8481 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 6, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 6 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:56.576635  8583 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:56.937026  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 696) took 2284 ms. Trace:
I20260709 13:02:56.937388  8583 rpcz_store.cc:276] 0709 13:02:54.652866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.417013 (+764147us) service_pool.cc:224] Handling call
0709 13:02:56.936957 (+1519944us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:56.742437  8654 rpcz_store.cc:269] 0709 13:02:52.966017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.966174 (+   157us) service_pool.cc:224] Handling call
0709 13:02:56.323533 (+3357359us) spinlock_profiling.cc:243] Waited 43.8 ms on lock 0x7b5400424854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:56.323758 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100732288}
I20260709 13:02:56.786365 12255 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.769209  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:56.945158  8730 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:56.790329  8657 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:56.791975 12171 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:56.797986  8697 connection.cc:256] RPC call timeout handler was delayed by -1.335850230999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
I20260709 13:02:56.953532  8481 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:56.802270  8576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2488) took 2186 ms (client timeout 1979 ms). Trace:
W20260709 13:02:56.955873  8576 rpcz_store.cc:269] 0709 13:02:54.615910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.681363 (+ 65453us) service_pool.cc:224] Handling call
0709 13:02:56.802059 (+2120696us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54000b6154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:56.802170 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3866199296}
I20260709 13:02:56.802356 12224 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.802970 11886 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:56.619496  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 668) took 3188 ms. Trace:
I20260709 13:02:56.958587  9328 rpcz_store.cc:276] 0709 13:02:53.431176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.431998 (+   822us) service_pool.cc:224] Handling call
0709 13:02:55.536127 (+2104129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.601018 (+1064891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.619304 (+ 18286us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b5400518450. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:56.619428 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4678125440}
I20260709 13:02:56.230435  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 881) took 2050 ms. Trace:
I20260709 13:02:56.623790  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:56.801554  9329 rpcz_store.cc:269] 0709 13:02:54.721420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.607328 (+1885908us) service_pool.cc:224] Handling call
0709 13:02:56.617550 (+ 10222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.091202  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.092341  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.791777  8724 rpcz_store.cc:276] 0709 13:02:53.549737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.550297 (+   560us) service_pool.cc:224] Handling call
0709 13:02:53.550655 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.299614 (+748959us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.440951 (+1141337us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b5400064d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:55.441118 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1617132416}
I20260709 13:02:56.737478 12349 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.799909 12345 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.805125  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:56.807500 12338 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.802194  9330 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:56.809557 12265 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.812044 12382 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.812290 12260 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:56.812482  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.46954e+06us
I20260709 13:02:56.812994 12228 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:56.810631 12353 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.110199 12353 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.111375  8730 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:56.811087 12335 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:56.812342 12323 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.810089 12154 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:56.814095 12332 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.113667 12332 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.115835 12332 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.821707  8727 raft_consensus.cc:2376] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8. Re-sending same reply.
I20260709 13:02:56.821834  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:56.629001  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08003280a0 after lost signal to thread 7484
I20260709 13:02:56.823184  8729 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:56.826519  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:56.821204 12158 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.820942 12358 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:56.832499 12340 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:56.716545 12339 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:56.837698  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 199890us
I20260709 13:02:56.840106  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:56.843747  9322 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:56.789309 12175 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:56.855638  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.126802  8582 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:57.118476  8581 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:56.855463  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:56.872895  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 129052us
I20260709 13:02:57.129169  8579 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:56.874004  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110333us
I20260709 13:02:56.875300  8651 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:56.694371  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 893) took 2510 ms. Trace:
I20260709 13:02:56.855994 12384 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:56.911381  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 130549us
I20260709 13:02:56.933125  9324 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:56.943832 12218 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:56.845193 12383 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.954223  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:56.799847  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:57.132907  9325 rpcz_store.cc:276] 0709 13:02:54.184156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.217068 (+ 32912us) service_pool.cc:224] Handling call
0709 13:02:54.223939 (+  6871us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.680272 (+2456333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.694068 (+ 13796us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b5400519354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:56.694304 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5646512384}
I20260709 13:02:56.845057 11704 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:56.974046  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.139730  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.140377  9325 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:56.850157 12114 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:56.985459  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:56.983716  8481 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:56.992601  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:57.144208  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 151662us
I20260709 13:02:57.144901  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:57.146626  8579 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:56.934667 12092 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.002085  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:57.002924  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.005674 11886 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:57.067205 12131 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:56.987680 12224 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.004482 12268 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.008759 12363 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:56.953692 12341 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:57.075519 12392 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.091s	user 0.005s	sys 0.013s
I20260709 13:02:57.074292 12376 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.077199 11955 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.078037 12334 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.072032 12163 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.153970 12340 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:57.093288  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.094575  9331 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:57.216214  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.094859  9327 rpcz_store.cc:276] 0709 13:02:54.179630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.181852 (+  2222us) service_pool.cc:224] Handling call
0709 13:02:54.182206 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.479981 (+297775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.230113 (+1750132us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54007ab554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:56.230346 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":677731200}
I20260709 13:02:57.100389  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.103737 12349 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.102284  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:57.219807  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 117598us
I20260709 13:02:57.109086 12345 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:57.117796  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08002cf080 after lost signal to thread 8516
I20260709 13:02:57.116757  8732 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:56.832319 12304 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.113209 12323 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.235004 12384 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.133442 12353 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.134042  9322 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:02:57.236019  9322 raft_consensus.cc:606] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:57.153846 12158 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 13:02:57.237223  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.237845 12158 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:57.238065  9322 raft_consensus.cc:2410] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:57.238207 12158 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:57.239010  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.239076 12158 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.240244  9324 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:02:57.240303 12158 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:57.241107  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 896) took 2761 ms. Trace:
W20260709 13:02:57.137364  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 337525us
I20260709 13:02:57.242667  9324 rpcz_store.cc:276] 0709 13:02:54.479244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.239668 (+2760424us) service_pool.cc:224] Handling call
0709 13:02:57.240004 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:57.240827 (+   823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.241047 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:57.138262  8696 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.979s (SENT)
W20260709 13:02:57.245903  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 107649us
W20260709 13:02:57.247481  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.984036 12290 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:57.150465  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 196325us
I20260709 13:02:57.250872  8697 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:57.252563  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:57.151443  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:02:57.151816  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 149804us
I20260709 13:02:57.149356  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:57.153658  8725 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.253876  8697 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:57.133443 12294 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.157121  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:57.255461  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 98410.3us
W20260709 13:02:57.255622  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.136749 12342 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.152995 11797 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:57.275565 12112 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:57.275759 11797 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.276279 11797 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.276152 12112 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.205379  8733 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:02:57.206257  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.278178  9328 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:57.279270 12360 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:57.119202 12260 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.093823 12115 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:57.279899 12360 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.281946 12260 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.281956 12360 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.152619 12256 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:57.213304  8726 raft_consensus.cc:2376] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6. Re-sending same reply.
I20260709 13:02:57.106189 12265 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.217463  9329 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:57.156174 12382 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:57.218554  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.226235 12338 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:57.226613  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.217829 12340 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:57.229866  9352 debug-util.cc:398] Leaking SignalData structure 0x7b0800302420 after lost signal to thread 8608
I20260709 13:02:57.230080  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.142645 12154 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 13:02:57.207542 11843 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:57.133262  8625 leader_election.cc:336] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.568s (SENT)
W20260709 13:02:57.305346  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 172062us
W20260709 13:02:57.306447  8625 leader_election.cc:336] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.751s (SENT)
W20260709 13:02:57.153147  8466 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a0d703ee18464475a290425976246555 (127.7.78.65:46779) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:57.318235 12339 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.318979  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.319357 12265 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.319656  8583 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:57.320647 12338 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:57.320797  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.321624 12338 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.322693  9330 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
W20260709 13:02:57.254652  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 105365us
I20260709 13:02:57.315872  9328 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [3.6-7.8]
I20260709 13:02:57.154673  8481 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:57.329867 11843 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:57.331292 12268 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:02:57.318264  8466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 165153us
I20260709 13:02:57.331722 12268 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:57.333639 12268 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.256786  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:57.335803  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 79080.2us
I20260709 13:02:57.336604  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 699) took 2608 ms. Trace:
I20260709 13:02:57.249706 12332 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.258556  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 898) took 2635 ms. Trace:
I20260709 13:02:57.267021  9324 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:02:57.269155  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800241c80 after lost signal to thread 7481
W20260709 13:02:57.348806  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5520 after lost signal to thread 7484
I20260709 13:02:57.347937  9322 rpcz_store.cc:276] 0709 13:02:54.623209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.239975 (+2616766us) service_pool.cc:224] Handling call
0709 13:02:57.257922 (+ 17947us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.258195 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:57.258239 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:57.258297 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:57.258337 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.258390 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.258497 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:57.349990  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800243d00 after lost signal to thread 8516
I20260709 13:02:57.350802  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:57.351789  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800256d40 after lost signal to thread 8608
W20260709 13:02:57.353117  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800070540 after lost signal to thread 8683
I20260709 13:02:57.271546 12383 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.273011  9326 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:57.273546 12343 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:57.274015 12358 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.274650 12323 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.274843  9325 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:57.275278 12224 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.279193 11900 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:57.153363 12362 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:57.285562  9327 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:02:57.228153  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.383173  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.324965 12384 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:57.299196  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2510) took 3063 ms (client timeout 2089 ms). Trace:
W20260709 13:02:57.301458  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08002cf4e0 after lost signal to thread 8683
W20260709 13:02:57.387689  8651 rpcz_store.cc:269] 0709 13:02:54.235498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.236985 (+  1487us) service_pool.cc:224] Handling call
0709 13:02:57.298979 (+3061994us) spinlock_profiling.cc:243] Waited 2.59 s on lock 0x7b540007eed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:57.299105 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5967759232}
W20260709 13:02:57.388741  9352 debug-util.cc:398] Leaking SignalData structure 0x7b0800320cc0 after lost signal to thread 8757
W20260709 13:02:57.389192  9352 debug-util.cc:398] Leaking SignalData structure 0x7b080035ea60 after lost signal to thread 8758
W20260709 13:02:57.390268  9352 debug-util.cc:398] Leaking SignalData structure 0x7b0800378960 after lost signal to thread 9355
I20260709 13:02:57.315867  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.317014  8582 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:57.390866  8578 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:57.391476  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.319778 11955 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.244853  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:57.391914  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.340209  8583 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:57.392566  8578 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.297915 12342 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.393126  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:57.134553  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.394491  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.395124 12342 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.282981 12360 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.395861  8651 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:57.395911 12360 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.340950  9329 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:02:57.398414 12360 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:57.356990  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08002dc840 after lost signal to thread 8757
I20260709 13:02:57.264484 12353 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:57.335058 12268 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:57.399716  8657 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:57.283422 12163 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:57.283771 12350 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:57.400902 12350 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.337082  8575 rpcz_store.cc:276] 0709 13:02:54.728540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.366749 (+1638209us) service_pool.cc:224] Handling call
0709 13:02:56.367045 (+   296us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.720412 (+353367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.720445 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:56.720922 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 13:02:56.849673 (+128751us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:57.005907 (+156234us) log.cc:844] Serialized 14 byte log entry
0709 13:02:57.006296 (+   389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.082497 (+ 76201us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b540002dd54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:57.082554 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.309456 (+226902us) raft_consensus.cc:1685] finished
0709 13:02:57.336400 (+ 26944us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.336537 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26266,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":232,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":186,"spinlock_wait_cycles":811575552,"thread_start_us":276214,"threads_started":2,"wal-append.queue_time_us":130927}
I20260709 13:02:57.402087  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:57.402918  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003f0c20 after lost signal to thread 8758
I20260709 13:02:57.298353 12382 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:57.404458  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800359c20 after lost signal to thread 9355
W20260709 13:02:57.404837  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003a6e20 after lost signal to thread 11569
I20260709 13:02:57.302770 12260 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:57.406232  8582 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:57.297541 12352 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.367038 12131 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:57.295104  8725 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:02:57.384087  9330 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:57.384486  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.390027  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.464149  9325 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 13:02:57.384801 12391 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 13:02:57.402839  8575 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
W20260709 13:02:57.405090  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080005f620 after lost signal to thread 11704
W20260709 13:02:57.466238  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003a6e80 after lost signal to thread 12092
W20260709 13:02:57.466617  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800352fc0 after lost signal to thread 12112
W20260709 13:02:57.466872  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800359cc0 after lost signal to thread 12130
I20260709 13:02:57.466723  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.467659  8575 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:57.377779  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:57.468873  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 91183.9us
I20260709 13:02:57.469775  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:57.472247  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.473456  8575 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:57.474889  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.353699  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:02:57.303673 12256 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.415616  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.309705 12340 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:57.446209  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 201434us
I20260709 13:02:57.412870  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.283998 12294 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.375089 12092 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:57.454713  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.387578 12384 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.365957 12324 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.467247  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003a6d20 after lost signal to thread 12131
I20260709 13:02:57.409083  8578 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:57.477037  8654 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:02:57.361491 12339 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.104s
I20260709 13:02:57.482550  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.401329 12350 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:57.493237  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.459321  8481 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 5, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 5 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:02:57.401515 11900 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:57.518846  7486 debug-util.cc:398] Leaking SignalData structure 0x7b080000ff40 after lost signal to thread 8549
I20260709 13:02:57.512142  8582 raft_consensus.cc:2376] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8. Re-sending same reply.
I20260709 13:02:57.463187 12131 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:57.481276  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 127598us
I20260709 13:02:57.462787 12163 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:57.508728  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003a6de0 after lost signal to thread 12154
I20260709 13:02:57.483486 12342 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:57.513163  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.539744  8577 raft_consensus.cc:2376] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7. Re-sending same reply.
I20260709 13:02:57.462288 11704 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:57.484834 12392 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 13:02:57.555011  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.386509 12228 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:57.373715 12332 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.377684 12363 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.119s
I20260709 13:02:57.446385 12269 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20260709 13:02:57.477641 12352 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.565495  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.566115  8577 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:57.531378 12131 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.480630 12256 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.382928 12343 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.531003  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:57.532367  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800329e00 after lost signal to thread 12163
W20260709 13:02:57.578502  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080005fb20 after lost signal to thread 12215
I20260709 13:02:57.578636  8654 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 13:02:57.578999  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800352f00 after lost signal to thread 12219
W20260709 13:02:57.579353  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800392ee0 after lost signal to thread 12228
W20260709 13:02:57.579725  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800352840 after lost signal to thread 12253
W20260709 13:02:57.580050  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080019e980 after lost signal to thread 12256
W20260709 13:02:57.580386  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08002fcfa0 after lost signal to thread 12261
W20260709 13:02:57.580785  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800328c40 after lost signal to thread 12265
I20260709 13:02:57.452703 12382 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:57.361792 11886 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.532655  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:57.584235  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51671.7us
I20260709 13:02:57.495698 12092 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.501514 12360 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:57.534439  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:57.542167  8730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:02:57.586988  8581 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:57.511586 12384 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.547663  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.560986 12342 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:57.591272  8657 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
W20260709 13:02:57.593767  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59414.4us
I20260709 13:02:57.503314 12362 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:57.555981  8582 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:57.596616  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.597299  8582 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:57.554337 12323 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:02:57.557443  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.588395 12329 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:57.558995  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:57.559346  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.603775  8659 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.465049 12260 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.573077  8650 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:02:57.524828 11843 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:57.535478 12290 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.581736  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080034f780 after lost signal to thread 12311
I20260709 13:02:57.543537 12341 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.543107 11704 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:57.599408 12403 raft_consensus.cc:2764] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:57.560216 12228 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.521912 12324 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:57.590095  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.542057 12154 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 13:02:57.592783 12342 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.598033  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:57.588712 12092 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:57.590169 12360 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.520520 12171 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:57.580574 12383 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
W20260709 13:02:57.662418 12402 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 13:02:57.662667 11900 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:57.620438  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080036b080 after lost signal to thread 12316
W20260709 13:02:57.663653  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080036b880 after lost signal to thread 12317
W20260709 13:02:57.663947  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c42e0 after lost signal to thread 12318
W20260709 13:02:57.664180  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800287440 after lost signal to thread 12319
W20260709 13:02:57.664389  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080031bbc0 after lost signal to thread 12320
W20260709 13:02:57.664587  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800275d40 after lost signal to thread 12322
W20260709 13:02:57.664777  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800269aa0 after lost signal to thread 12323
W20260709 13:02:57.665059  8680 debug-util.cc:398] Leaking SignalData structure 0x7b0800385f60 after lost signal to thread 12334
W20260709 13:02:57.665303  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c8720 after lost signal to thread 12337
W20260709 13:02:57.665529  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c8800 after lost signal to thread 12338
W20260709 13:02:57.665720  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c87e0 after lost signal to thread 12339
W20260709 13:02:57.665910  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c8840 after lost signal to thread 12340
I20260709 13:02:57.666100 12343 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.659096 12352 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.666538  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.667660 12269 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:57.668094 12269 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:57.668859 12114 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.660970 12360 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:57.629204  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:57.669708 12175 consensus_meta.cc:287] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.000s
I20260709 13:02:57.670112  8650 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:57.378423 12335 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.670368  9323 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:57.596073 12362 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.670666 12335 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.672447 12362 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.672976 12335 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:57.654603  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 390571us
I20260709 13:02:57.674212  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:02:57.674971 12403 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.675419 12403 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.676852 11955 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.648864 12341 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:57.677485 12403 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.619489 12329 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:57.677671 12341 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:57.678610 12341 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.679204 12158 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:57.678731 12329 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.679652 12158 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:57.679914 12341 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.680087 12158 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:57.604662 12339 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 13:02:57.646369 11704 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:57.657763  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62282.4us
I20260709 13:02:57.668932 12269 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.636021  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:57.646966  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.684619  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:57.685233  8655 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:57.625945  8659 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:57.677143 12131 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.660804 12345 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:57.690933  9330 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:57.703824 12403 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:57.704909 12131 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.654508 12349 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.707357  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.708957  8732 pending_rounds.cc:85] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:02:57.713009  8732 pending_rounds.cc:107] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260709 13:02:57.713382  8732 raft_consensus.cc:2889] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:57.713559  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.710733  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.712320  8733 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:57.630772 11843 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.631814 12290 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.606868 12163 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.686116 12304 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.706506 12154 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:57.702637 12345 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.718544 12269 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.702884 11955 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.746423 12154 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.750378 12345 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:57.754395 12405 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 13:02:57.723093 12349 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.754454 12290 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.770372 12349 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.779211 12382 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.785357 12350 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.787261 12382 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.787513 12350 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.790959 12360 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.789367 12384 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:57.793077 12360 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.793140 12384 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:57.528265  8481 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:57.533916  8575 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:57.587368  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.832576  8577 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:57.836206  8582 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:02:57.666121  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c8820 after lost signal to thread 12342
W20260709 13:02:57.838225  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08002d8280 after lost signal to thread 12344
I20260709 13:02:57.584930  8583 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:02:57.838728  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c86e0 after lost signal to thread 12345
W20260709 13:02:57.839109  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003c87a0 after lost signal to thread 12348
W20260709 13:02:57.839442  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080025a000 after lost signal to thread 12349
W20260709 13:02:57.839726  8680 debug-util.cc:398] Leaking SignalData structure 0x7b08003356c0 after lost signal to thread 12350
W20260709 13:02:57.839991  8680 debug-util.cc:398] Leaking SignalData structure 0x7b080033d140 after lost signal to thread 12351
I20260709 13:02:57.846756  8730 raft_consensus.cc:606] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:57.517771  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.849295  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:57.850040  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 378626us
I20260709 13:02:57.850580  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.851478  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.848829  8548 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:57.854426  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.856055  8652 raft_consensus.cc:2410] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:02:57.856206  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:57.851594  8578 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:57.857287  8654 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.858443  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.859243  8578 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.854344  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.862321  8650 raft_consensus.cc:2410] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:57.862490  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.864161  8655 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:57.838083  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.867645  8657 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:57.849941  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.834782  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:57.879748  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.880729  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.879748  8577 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:57.881124  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.882800  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.883750  8584 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:57.884213  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.884899  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.882351  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.886102  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.889060  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.890564  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.890923  8726 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:57.882267  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:57.891028  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.891842  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:57.892014  8656 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:57.892272  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.892843  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.893703  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.894904  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:57.886830  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.908524  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:57.909830  8733 raft_consensus.cc:2368] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9.
I20260709 13:02:57.732105 11797 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:57.911183  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:57.912717  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:57.913786  9325 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:57.914095  8724 raft_consensus.cc:2376] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6. Re-sending same reply.
I20260709 13:02:57.913340  8481 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 4, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:57.915637  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:57.912955 11797 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:57.913362  8582 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:57.912717  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.937361  8655 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:57.938153  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:57.938742  8659 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.939632  8724 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:57.941838  8576 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
W20260709 13:02:57.943264  9326 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 891) took 4979 ms (client timeout 1820 ms). Trace:
W20260709 13:02:57.943792  9326 rpcz_store.cc:269] 0709 13:02:52.963704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.963820 (+   116us) service_pool.cc:224] Handling call
0709 13:02:57.934552 (+4970732us) spinlock_profiling.cc:243] Waited 4.27 s on lock 0x7b5400552654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:57.943008 (+  8456us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9828450688}
I20260709 13:02:57.944439  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.763056 12154 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:57.946411  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.947352  8726 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:57.947650  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.948524  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.950171  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.916276  8481 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:57.952292  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.953235 12112 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:57.953639  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.955401  8577 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:57.956521 12132 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:57.958434  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:57.959337  9324 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:57.957087  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:57.948702  8580 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.953310  8582 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:57.964898  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.966300  8733 raft_consensus.cc:2410] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:57.961243  8729 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:57.956992 12260 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:02:57.975440  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 87248.5us
I20260709 13:02:57.976812 12294 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.977344  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:57.978048  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:57.979281  8584 raft_consensus.cc:2410] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:02:57.983872  8584 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:02:57.984782  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.986217  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.986580  8657 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:57.987267  8729 raft_consensus.cc:2410] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:57.987767  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:57.988171  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:57.988873  8729 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:57.989019  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:57.957131 12132 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:57.990821  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:57.991372 11843 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.991669  9330 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.993578  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:57.995095 12332 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:57.997748 11955 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:58.003268  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.005766 12360 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:58.006412 12360 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.006910 12360 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:58.005461  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.011487  8654 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:02:58.015009  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 58959us
I20260709 13:02:58.016446  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.017593  8726 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:58.019096  8726 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:58.014627  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.021209  8726 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 13:02:58.021476  8575 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:58.023226  8575 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 13:02:58.026042  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2079) took 1401 ms. Trace:
I20260709 13:02:58.026561  8725 rpcz_store.cc:276] 0709 13:02:56.624792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.816576 (+191784us) service_pool.cc:224] Handling call
0709 13:02:58.025664 (+1209088us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400338954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:58.025850 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":773022464}
I20260709 13:02:58.029605  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.031019  8729 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:58.031459  8727 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:58.038636  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.040239  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.046118  8726 pending_rounds.cc:85] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:02:58.044449 12132 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.018069 12268 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:58.054158  9324 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:58.056648  9325 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:57.979219 12412 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.059060 12412 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.059492  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:58.046481  8726 pending_rounds.cc:107] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 13:02:58.067595  8726 raft_consensus.cc:2889] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:58.068627  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:58.071357  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.076666  8578 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:02:58.082526  8733 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:58.083299  8724 raft_consensus.cc:2410] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:02:58.083448 12342 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:58.072137  8652 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:58.084141  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:58.085330  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.085832 12382 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.086338 12382 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.086691 12382 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.086930  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:58.089625 12290 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 13:02:58.090258  8652 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:58.094184  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:58.097714  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.074913 12294 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.070724 12412 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:58.111222  8580 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:58.114035  8575 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:58.115320  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.120533  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:58.121471  8654 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:58.125669 12354 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:58.126544 12354 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:58.127911 12349 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:58.128093 12354 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.129181  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:58.130110  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 55374.8us
I20260709 13:02:58.130890 12376 raft_consensus.cc:2764] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:02:58.135196  8584 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:58.136240  8656 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:58.138911 12349 consensus_queue.cc:1243] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:02:58.139595  8652 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:58.139976 12132 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:58.141850  8656 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:02:58.146126  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 93517.2us
I20260709 13:02:58.152283  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:58.157914  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98655.8us
I20260709 13:02:58.097419  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:58.167369  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.162635  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:58.112409 12294 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.184090  8659 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:58.161842 12268 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:58.197193  8657 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:58.187834 12294 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.226788  9325 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:58.227336  9325 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:58.228340  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:58.229943  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.230456  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.230743  8654 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:58.229290  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:58.244935 12345 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.247938 12345 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:58.250152 12360 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.251637  8731 raft_consensus.cc:2368] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7.
I20260709 13:02:58.255429  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:58.255301  9325 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.256982  9325 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:58.257076  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:58.258208 12268 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.259433 12304 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:58.259950 12304 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:58.260456 12304 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:58.265213  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.265566  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.265823  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:58.266150  8732 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:58.266357  8659 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:58.267132  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.267861  8584 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20260709 13:02:58.278355  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.258280  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:58.282819  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 84610.6us
I20260709 13:02:58.286396  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.279186  8727 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
I20260709 13:02:58.253137 12360 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:58.259156 12163 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:58.294842  9325 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 13:02:58.295885  9325 pending_rounds.cc:85] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 7
I20260709 13:02:58.296939  9325 pending_rounds.cc:107] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 13:02:58.297554  9325 raft_consensus.cc:2889] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:58.240813 11797 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:58.300307  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 106382us
I20260709 13:02:58.304759  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.312191  8577 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:58.305732  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.321141  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:58.322184  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.324222  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.327188  8580 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:58.332594  8730 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:58.333595  8726 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:58.334712  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.336206  8726 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:58.337023 12158 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.656s	user 0.015s	sys 0.002s
I20260709 13:02:58.337711 12158 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.340634 12158 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:02:58.340843 12426 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 13:02:58.352778  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:58.357285  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.386885  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.366252  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:58.396173  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 141735us
I20260709 13:02:58.360438  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.398118  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.304270 12354 raft_consensus.cc:2764] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260709 13:02:58.372611  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54441.2us
I20260709 13:02:58.373236  8727 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:02:58.407979  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:58.381203  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.364914  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:58.317759 12163 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:58.402100 12338 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:58.403059 12329 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:58.403153  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 710) took 1077 ms. Trace:
I20260709 13:02:58.403841 12335 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:58.311353 11797 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:58.441691 11797 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.443488 12353 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:58.422964  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70387.8us
I20260709 13:02:58.401044  9330 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:58.431845  8729 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:58.433640  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.434191  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.438627  8732 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:58.364657 12324 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:58.434913  8575 rpcz_store.cc:276] 0709 13:02:57.326223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.023960 (+697737us) service_pool.cc:224] Handling call
0709 13:02:58.047671 (+ 23711us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:58.116700 (+ 69029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.402896 (+286196us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b5400343fd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:58.403075 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53566208}
I20260709 13:02:58.448733  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:58.460758  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 159086us
I20260709 13:02:58.458168  8730 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:58.462713  8733 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:58.463336  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.465590  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.469532  8726 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:58.466070  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.384768 12358 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 13:02:58.474189  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.476217  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.476747  8727 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:58.479570  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.406582 12345 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 13:02:58.479249 12329 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:58.481999 12171 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:02:58.482282 12256 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.820s	user 0.018s	sys 0.003s
I20260709 13:02:58.483130  8654 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 13:02:58.488548  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.463196 12290 raft_consensus.cc:2764] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:02:58.467818 12338 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:58.486565  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.495756  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:58.489070  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56829.6us
I20260709 13:02:58.500293  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.448278 12383 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 13:02:58.492242 12171 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:58.525816 12163 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.514824  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:58.534166  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.535503 12329 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.536276  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.536722  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.537029  8580 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:58.543426 12312 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:58.543988 12312 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:58.551285  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.551401 12353 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:58.552024 12353 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:58.552323 12304 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.532564  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:58.554941  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.530515  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:58.498611 12256 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:58.558673  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45160.2us
I20260709 13:02:58.552040  8583 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:58.563170  8575 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:58.562107 12228 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.903s	user 0.009s	sys 0.005s
I20260709 13:02:58.564967  8659 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:02:58.566114 11704 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.829s	user 0.018s	sys 0.000s
I20260709 13:02:58.566598 11704 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.568370  8658 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
W20260709 13:02:58.568900 12092 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.879s	user 0.008s	sys 0.004s
I20260709 13:02:58.569103 11704 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:58.583503 12324 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:02:58.584437 12382 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.585155 12324 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.585527  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2495) took 1275 ms. Trace:
I20260709 13:02:58.585815 12324 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.585908  8576 rpcz_store.cc:276] 0709 13:02:57.309929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.310048 (+   119us) service_pool.cc:224] Handling call
0709 13:02:57.310342 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.942434 (+632092us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.585161 (+642727us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b54000b6154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:58.585446 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393472896}
W20260709 13:02:58.587720 12269 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.835s	user 0.002s	sys 0.010s
I20260709 13:02:58.588184 12324 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:58.589943  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:58.591475 12338 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:58.595019  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.595757 12335 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.596283 12335 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:02:58.596765 12335 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.598541  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2525) took 1144 ms. Trace:
I20260709 13:02:58.598985  8653 rpcz_store.cc:276] 0709 13:02:57.454235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.454424 (+   189us) service_pool.cc:224] Handling call
0709 13:02:58.598177 (+1143753us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54001310d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:58.598341 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1069494656}
I20260709 13:02:58.600663  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2490) took 1293 ms. Trace:
I20260709 13:02:58.601929 12354 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:58.601097  8579 rpcz_store.cc:276] 0709 13:02:57.306692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.308655 (+  1963us) service_pool.cc:224] Handling call
0709 13:02:57.318599 (+  9944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.826074 (+507475us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:57.827068 (+   994us) log.cc:844] Serialized 14 byte log entry
0709 13:02:57.933729 (+106661us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:57.933800 (+    71us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:57.933863 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.600278 (+666415us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54003a3c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:58.600321 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:58.600501 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":103643,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":3717,"raft.queue_time_us":80609,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":250,"spinlock_wait_cycles":1164774528,"thread_start_us":71272,"threads_started":1,"wal-append.queue_time_us":422526}
I20260709 13:02:58.602385 12354 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:58.602952 12340 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.065s	user 0.010s	sys 0.011s
I20260709 13:02:58.603474 12340 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.605026  8581 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:58.605733 12349 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:58.607034  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2501) took 1294 ms. Trace:
I20260709 13:02:58.569389 12092 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.607443  8581 rpcz_store.cc:276] 0709 13:02:57.312121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.972888 (+660767us) service_pool.cc:224] Handling call
0709 13:02:58.606827 (+633939us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54001680d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:02:58.606971 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1380777856}
I20260709 13:02:58.600528  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.609107 12092 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:58.609886  8651 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:02:58.611588  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2523) took 1198 ms. Trace:
I20260709 13:02:58.611905  8651 rpcz_store.cc:276] 0709 13:02:57.413358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.413518 (+   160us) service_pool.cc:224] Handling call
0709 13:02:57.413859 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:58.610332 (+1196473us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.611306 (+   974us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540013a454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:02:58.611516 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2749238016}
I20260709 13:02:58.593083 12358 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:58.613776 12358 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:58.614224 12358 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:58.616596 12115 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:58.617022 12115 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:58.603385 12354 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.618674  8733 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:02:58.619426  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48867.3us
W20260709 13:02:58.624967  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.618803  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:58.630324  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53309.5us
W20260709 13:02:58.629777 12175 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
W20260709 13:02:58.634320  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 144352us
I20260709 13:02:58.636263 12304 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:58.635941 12115 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.637250 12115 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:58.563586 11797 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260709 13:02:58.545148 12312 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.651531 12256 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:58.653173 12349 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:58.659050  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:58.579900 12228 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.663323  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.664273  8583 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:02:58.665880  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:58.673208  8550 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.533s (SENT)
I20260709 13:02:58.673249  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:58.675014 12304 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:58.675680 12304 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.676036 12304 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:58.675912 12383 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:58.676151  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:58.677845 12383 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.678310 12383 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:58.678752 12383 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.680764  9324 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:02:58.684525 12294 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.642285 12329 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:02:58.686115  9323 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:02:58.688107 12132 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.548s	user 0.013s	sys 0.000s
I20260709 13:02:58.688264 12340 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:58.678198  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:58.688396 12383 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.690064  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:58.691936  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:58.693449  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 192546us
I20260709 13:02:58.731424  8578 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:58.733946  8724 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:58.733744  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:58.736232  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:58.737448  9325 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:02:58.741535  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51474.9us
W20260709 13:02:58.743048  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.747437  9324 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:58.748546  8580 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:02:58.698726 12171 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.739565 12175 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:58.738312 12324 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.743091 12354 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:58.761492 12154 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.790s	user 0.011s	sys 0.004s
I20260709 13:02:58.761232 12132 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:02:58.762048  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84044.7us
I20260709 13:02:58.762300 12132 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:58.762827 12132 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:58.761955 12154 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.750633 12092 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:58.763170 12317 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.773665 12358 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.774381 12382 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:58.776134 12358 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:58.776953 12294 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:58.773680 12317 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.778014 12317 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.760648 12340 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:02:58.780082  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.753720 12175 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.785115 12175 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:58.790231  8726 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:58.791026 12154 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:02:58.792451 12441 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.231s	user 0.001s	sys 0.001s
I20260709 13:02:58.793922  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:58.803673  8575 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:58.770393 12324 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:58.801481  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 109637us
I20260709 13:02:58.806026  9324 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:58.810761  8548 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:58.806094 12312 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:58.822068  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:58.826558  8582 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:58.822743  9297 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.166s (SENT)
I20260709 13:02:58.806880 12092 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:58.806607 12340 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:58.831964  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:58.793241 12317 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.807799 12154 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.840806  8575 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:02:58.825752 11797 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:02:58.797224 12304 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.837033 12154 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.854763 12383 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:58.855149 12092 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.857131 12335 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.858665  9330 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:58.858841 12304 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:58.861685 12448 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:58.862182  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:58.862432 12448 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:58.862358 12353 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:58.863170  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 360857us
I20260709 13:02:58.864003  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:58.863798 12448 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.864998 12353 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:58.866665 12269 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.841804 12358 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 13:02:58.868196 12342 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:58.868463 11843 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:02:58.882827  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 50668us
I20260709 13:02:58.851982 12154 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:58.885656  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.885531 11797 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.886147 12268 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.887266  8623 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:58.887861 12342 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.854790  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:58.901860  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:58.902472  8724 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:58.911693  8724 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:58.911924  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 92068.1us
I20260709 13:02:58.890960 11843 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:58.916038  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.917380  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:58.916148  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:58.908017 12342 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.880280 12269 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.921032  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:58.850793 12132 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:58.931041  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 43852us
I20260709 13:02:58.957657  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.957991  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.922351 11843 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.922978 12218 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:58.959759  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:58.959973  8726 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:58.961050  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.961825  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:58.957019 12445 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 13:02:58.964427  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:58.985496  8733 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:58.963574  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:58.932852 12342 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.982887  8731 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:58.984822  8730 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:02:58.984864  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.963575  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.960323 12294 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.987169  8651 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:58.987258  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.913133 12412 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 13:02:58.930080 12115 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.987185  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.982649  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.960979 12358 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:58.996359  8728 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:02:58.997937  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:58.997830  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.998585  8577 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:58.999076  8656 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:58.999379  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.999985  8659 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:58.960088 12290 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:59.002544  8658 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:59.000844  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:58.963333 12163 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:59.009769  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:58.991014 12304 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:59.014438  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54303.1us
I20260709 13:02:59.015443  8657 raft_consensus.cc:2410] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:59.017275  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.017488  8652 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:02:59.018675 12338 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.019555 12338 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:59.019965  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 103943us
I20260709 13:02:59.021081  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:59.002916 12358 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.022249 12358 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.022374 12290 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:59.023535 12290 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:59.023906 12290 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.024238 12290 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.026636  8696 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:59.027640 12353 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:59.028316  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:59.029165 12158 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:59.030260 12353 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:58.930316 12340 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:58.961597 12268 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:58.992549 12115 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.040791  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.041692  8652 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:59.041985  8578 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:59.042467  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.043452  8728 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:02:59.048373  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:58.997495 12294 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.035796 12132 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.034192 12256 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:59.053305  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:59.054777  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.061816  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.062176  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.062867  8583 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:59.061861  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.064213  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.009918 12383 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:02:59.069881  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.070753  8578 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:59.073124  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.049834 12324 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:59.077692  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.078265  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.078435  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.079599 12171 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.080554  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.089990  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.092187  8658 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:59.097651  8582 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:59.099174  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:02:59.101877  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.104121  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:59.107267  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69861.3us
I20260709 13:02:59.108673  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.109768  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.110778  8659 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:59.118096  8578 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:59.119525  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.122155  8579 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.121203  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.122609  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.124716  8657 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:59.125420  8658 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:59.133807  8580 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:02:59.088791 12338 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.136858  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.045710 12163 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.052541 12175 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.144788  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:02:59.145432 12459 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
W20260709 13:02:59.146557 12454 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 13:02:59.146986  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.125460 12383 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:59.123711 12158 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.126573  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.149696  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.154433 12268 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:59.163509  8732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:02:59.166429  8732 raft_consensus.cc:606] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:02:59.166553  8658 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:59.166692 12158 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.170436  8579 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:02:59.148720 12269 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.160694  8729 raft_consensus.cc:2410] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 13:02:59.170701  8549 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.623s (SENT)
I20260709 13:02:59.178448  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.173357  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.180450  8659 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:02:59.173858  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 214140us
I20260709 13:02:59.135767  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:59.174055  8652 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:59.130522  8582 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:59.181582  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:02:59.182395  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 161360us
I20260709 13:02:59.179543  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.172180  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:02:59.189621  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:59.171474  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:02:59.206714  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.207669  8655 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:59.210214  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.208992  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.212920  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.177271 11797 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:59.218065  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.196471 12353 consensus_queue.cc:1243] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:02:59.211313  8732 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:02:59.218797  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.215350  8581 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:59.214188  8659 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:02:59.222497  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:59.231313  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59237.3us
I20260709 13:02:59.236506  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.237546  8659 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:59.219904 12256 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 13:02:59.238893  8659 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:02:59.239192  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:59.193922 12468 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:59.221323 12268 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:59.245942 12453 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:59.245800 12465 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 13:02:59.250810 12447 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:59.255563  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:59.274669  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.255600  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.282976  8650 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
W20260709 13:02:59.256742  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.286087 12158 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:59.286794 12158 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:02:59.276671  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 74983us
I20260709 13:02:59.287940 12158 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.288722  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:59.279595  8577 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:59.264654  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.291069  8582 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:59.293668  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.282905  8652 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
W20260709 13:02:59.292212  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 129129us
I20260709 13:02:59.305027  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:02:59.272770 12354 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.509s	user 0.007s	sys 0.002s
I20260709 13:02:59.292961 12115 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:59.272236 12290 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.309552 12294 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.313105  8583 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:59.253326 12335 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:59.253836 12353 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:02:59.270562 12175 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:59.321053  8582 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:59.263469 12163 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.257457 12304 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:59.361369 12304 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.291985 12447 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:02:59.363257  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:59.363175 12447 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.364364 12447 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.365677 12354 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:59.366415 12354 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.367745 12354 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.373378 12290 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:59.373556 12218 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:02:59.374631 12329 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:59.375173 12329 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:59.328804 12324 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 13:02:59.378057 12092 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:59.378597 12092 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.380652  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:59.381040  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.381373  8729 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:59.381771  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:59.327253  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:59.331779  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 76281.3us
I20260709 13:02:59.331956 12353 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.332602  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.390619  9324 raft_consensus.cc:2410] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:02:59.332906  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.333724 12171 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:59.334287 12268 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:59.335047  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46422us
I20260709 13:02:59.335112 12467 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:59.336447 12294 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:59.401741  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.338109 12342 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.343071  8725 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:02:59.346372 12228 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.348089  8656 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "081470ef13974fe497f88859ada43ffa"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38080
I20260709 13:02:59.329612 12338 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.359694 12317 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:59.412259  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.413885  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:59.359889 12358 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:02:59.360615  8550 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 2.043s (SENT) [suppressed 51 similar messages]
W20260709 13:02:59.388549  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.392520  8728 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:02:59.376647 12329 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.395828  8575 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
W20260709 13:02:59.397485  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70344.4us
I20260709 13:02:59.398530  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.257197 12468 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.405356  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.402751 12324 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:59.410763  8656 raft_consensus.cc:606] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Received request to transfer leadership
I20260709 13:02:59.457039  8656 raft_consensus.cc:614] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:59.416133  9322 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:02:59.395156 12163 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:59.462957  8651 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:02:59.412765  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.443127  9329 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:59.447242  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:59.448540  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:02:59.443660  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 83050.5us
I20260709 13:02:59.451500  9325 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:59.379846 12092 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.399643 12467 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:02:59.409910  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:59.444504  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:59.444121 12473 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.159s	user 0.004s	sys 0.000s
I20260709 13:02:59.413367 12294 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:02:59.469208  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:59.468886  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 81985.7us
I20260709 13:02:59.401505 11843 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.456621 12158 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:59.456604 12324 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.402902 12268 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.482685 12479 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:59.487425  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 84264.6us
I20260709 13:02:59.490880  8733 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:59.473179 12467 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:59.418546 12338 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.476596 12465 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:59.498507  8657 raft_consensus.cc:2368] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9.
I20260709 13:02:59.480010 12329 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:59.504801  9331 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:59.485136 12324 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:59.509658  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:59.478513 12383 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.490986 12455 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:59.488713 12092 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:59.532423  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 396721us
I20260709 13:02:59.480149 12175 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:02:59.492192 12158 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.540059  9328 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:59.469309 12304 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.507553 12163 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.546916  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.556557 12479 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.561812  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.562602  9327 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:02:59.562443 12158 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.564100  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:02:59.565526  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 55991.2us
I20260709 13:02:59.566426  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:59.587438  8625 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:59.543139 12466 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:59.591440 12115 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:59.588261 12312 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:59.596832  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.607609  9325 raft_consensus.cc:2368] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7.
W20260709 13:02:59.597961 12478 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 13:02:59.598912 12477 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
W20260709 13:02:59.584523 12154 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.675s	user 0.008s	sys 0.004s
I20260709 13:02:59.602404  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.558058 12479 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.587291 12465 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 13:02:59.611058  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:02:59.613863 12474 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.328s	user 0.000s	sys 0.003s
W20260709 13:02:59.614041 12476 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.326s	user 0.003s	sys 0.000s
I20260709 13:02:59.639735  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.603472 12304 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:59.595170 12455 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:59.622393 12338 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.568176 11797 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 13:02:59.623041 12358 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:59.623617 12131 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:02:59.596870 12472 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 13:02:59.652029  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:02:59.630589  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 138097us
I20260709 13:02:59.624802 12468 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.626050  9326 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 5.
I20260709 13:02:59.635942  9297 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:59.638417  9322 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:59.639318  9328 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:59.657943  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.616804 12268 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.642302  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:59.602717 12163 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.606482 12353 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.606503 12312 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.624933 12290 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:02:59.633736  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46376.4us
I20260709 13:02:59.674916  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:59.653134  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:59.644362  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.652474  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.610088 12294 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.676659 12483 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.676767  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:59.677229 12483 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.640934 12484 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 13:02:59.640407 12466 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.679728  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:59.679880 12466 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.680197 12256 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:59.680938 12115 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:59.681347  8653 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:59.681667 12115 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:59.682117 12466 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.664705  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.682807 12455 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:59.683954  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48087.6us
I20260709 13:02:59.666252  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:59.671250  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.674010 12317 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:59.702843  9323 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:59.684087 12484 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:59.671475  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:59.681762 12480 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 13:02:59.666343  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.679565  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:02:59.636641 12154 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:02:59.707243  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56435.8us
I20260709 13:02:59.715677  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.674114 12312 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:59.722915  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:02:59.729714  8583 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:59.725333  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.725654  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:59.729573  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53386.5us
I20260709 13:02:59.731043  8652 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:59.675396 11843 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:02:59.679605 12483 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:02:59.734354  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.735291  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:02:59.706100 12484 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:02:59.735764  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.732075  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:02:59.742918  8575 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.738539  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.743666  8724 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:59.737367  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.740430  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.740839  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.646732 12383 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.745570  8650 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:02:59.755863  8655 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:02:59.755985  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:02:59.745714  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.740321  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.757968  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.759210  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:59.761292  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.762396  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.767815  8653 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:02:59.729007 12175 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:59.768652  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.769320  8727 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:02:59.769634  8580 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:59.770356  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.774712  8730 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.778270  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.777798  8731 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
W20260709 13:02:59.779616  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 94901.9us
I20260709 13:02:59.781240  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:02:59.798278  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.796567  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.785707  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.802462  8733 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:02:59.811151  8730 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:02:59.818812  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:02:59.823411  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:59.771765 12290 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:59.834717  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.835093  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.835829  9323 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.831000  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:02:59.832085  8575 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:02:59.833330  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:59.762171 12268 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:59.833925  8655 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:02:59.834019 12171 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:59.836526  8727 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:59.837348  8730 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:02:59.785332 12342 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.839057  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.843602  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:59.848868  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.856413  8582 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:59.836056 12358 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:59.857290  9328 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:59.789304 12324 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.834606 12175 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:02:59.845978 12228 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:59.885229  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 143357us
W20260709 13:02:59.889474 12491 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 13:02:59.884712  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:59.890368 12132 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.824s	user 0.012s	sys 0.000s
I20260709 13:02:59.890771 12496 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:59.891165  8732 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:02:59.891335 12492 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 13:02:59.891001  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:59.896631  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.898519 12154 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:02:59.847065 12354 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:59.880843  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:02:59.908972  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:02:59.909682  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 167582us
W20260709 13:02:59.910868  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.911320  8658 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:59.913518  8727 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:02:59.908777  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:02:59.887511 12447 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.523s	user 0.009s	sys 0.004s
I20260709 13:02:59.850775 12483 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 13:02:59.920389  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:02:59.922792 12175 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:02:59.927894  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.928920  9327 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 13:02:59.929147  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.929517  8697 leader_election.cc:400] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7.
I20260709 13:02:59.929911  8697 leader_election.cc:403] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:59.930141  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.930246  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.930542  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.931213  9331 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:02:59.931341  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.932085  9327 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:59.933010  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.946281  9325 raft_consensus.cc:2368] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:02:59.851051 12480 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 13:02:59.934202  9328 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:59.934880  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.936772  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:02:59.942031 12489 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 13:02:59.942394  9330 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:02:59.942539  8731 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:59.942726 12488 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 13:02:59.920102  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:02:59.933720  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:02:59.927449 12484 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:02:59.951277  9331 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:02:59.955219  9323 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:02:59.979815  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.980295  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:02:59.984490  8725 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:59.917303 12465 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:59.982594  9326 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:02:59.946568 12358 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:59.986497  8730 raft_consensus.cc:2376] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9. Re-sending same reply.
W20260709 13:02:59.988888  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 59669.5us
W20260709 13:02:59.991426  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:59.985648  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 104871us
I20260709 13:02:59.946146 12496 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:02:59.984076  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:02:59.984903 12455 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.003334  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.988037 12154 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7.
I20260709 13:02:59.993326 12358 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.010537  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:00.020004  8731 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:03:00.012187  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:00.016443  8725 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:02:59.946909 12228 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 13:03:00.017962  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:00.019104  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:02:59.985564 12354 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:03:00.025951  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:00.025986  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.028959  8575 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:00.025055  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:00.030303  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:00.031932  8652 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:00.032884  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:00.032889  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:00.023018 12358 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:00.057329  8731 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:00.068738  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:00.065847  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:00.032990 12485 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:00.074069  8584 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:00.076713  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:00.074790  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.077423  8575 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:00.020843 12349 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:03:00.078862  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:00.057475 12492 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:00.019524 12455 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:00.068387 12092 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.512s	user 0.013s	sys 0.003s
W20260709 13:03:00.081683  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 69570.3us
I20260709 13:03:00.082734  8659 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:00.096957  9329 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:00.084846 12353 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:00.079627  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:00.096688  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:00.097990  8580 raft_consensus.cc:2410] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:03:00.098347  8728 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 13:03:00.088897  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:00.079243  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:00.126693  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:00.098739  8728 pending_rounds.cc:85] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 7
I20260709 13:03:00.089639 12484 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:00.091248 12496 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:00.092752  9326 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
W20260709 13:03:00.093627  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:00.073026 12342 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:00.083871 12499 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.119s	user 0.001s	sys 0.001s
I20260709 13:03:00.101207  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.079551  8654 raft_consensus.cc:2410] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 13:03:00.088594 12467 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.555s	user 0.005s	sys 0.005s
I20260709 13:03:00.171236  8652 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 13:03:00.177656  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 98501.7us
I20260709 13:03:00.126485 12329 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.130095  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:00.163869  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67242.8us
I20260709 13:03:00.081063 12497 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:00.095404 12486 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:00.168664  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:00.171346  8580 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 13:03:00.128587  8728 pending_rounds.cc:107] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 13:03:00.173024  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:00.182127  8580 pending_rounds.cc:85] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 7
I20260709 13:03:00.179358  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.182415  8728 raft_consensus.cc:2889] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:00.182137  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84162.9us
I20260709 13:03:00.183564  8659 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:00.179435  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:00.128521 12353 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:00.132612 12335 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:00.184895  8725 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:00.185423  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:00.182202  8653 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:00.182873  8580 pending_rounds.cc:107] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 13:03:00.192528  8580 raft_consensus.cc:2889] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:00.182830  8575 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:00.193212  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.193956  8659 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:00.195082  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:00.188666 12502 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 13:03:00.197553  8654 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:00.198941  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:00.199867  8584 raft_consensus.cc:2410] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:00.203660  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:00.204658  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:00.180769 12447 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:00.421890  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.434702  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:00.435546  8654 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:00.180819 12497 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:00.447427  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:00.470227  8653 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
W20260709 13:03:00.479969  9296 leader_election.cc:336] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.664s (SENT)
I20260709 13:03:00.481021  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:00.485526  8584 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:00.486095  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65069.2us
I20260709 13:03:00.177655 12342 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:00.561105  8584 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
W20260709 13:03:00.572486  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 378916us
I20260709 13:03:00.180214 12467 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:00.185948 12329 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:00.500211 12228 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:03:00.584921  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 397978us
I20260709 13:03:00.555702 12131 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.544981 12269 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:00.527275  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:00.510161  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.651160  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.530771  8575 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:00.634229  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:00.689440  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:00.710458  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 127063us
I20260709 13:03:00.196743 12486 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:00.437033 12499 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 13:03:00.685086  8658 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:00.720057  8654 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
W20260709 13:03:00.720274  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 98867.5us
I20260709 13:03:00.765331  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 714) took 1040 ms. Trace:
I20260709 13:03:00.502672 12497 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:03:00.752454  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 128504us
I20260709 13:03:00.774673  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:00.753635  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.765977  8732 rpcz_store.cc:276] 0709 13:02:59.724644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.724778 (+   134us) service_pool.cc:224] Handling call
0709 13:02:59.725085 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:59.891560 (+166475us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.765075 (+873515us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b5400417154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:00.765250 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381121280}
I20260709 13:03:00.776185  8657 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:00.759066  8659 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:00.778858  8650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.076s	sys 0.108s
W20260709 13:03:00.779299  8650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.076s	sys 0.108s
W20260709 13:03:00.743037  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53730.4us
I20260709 13:03:00.791265  8659 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:00.795619  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:00.711875 12269 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:00.802067  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:00.681514 12131 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:00.785135 12487 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:00.803056 12485 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:00.809780 12485 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:00.810417 12485 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:00.813794 12514 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.814442 12514 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:00.815771 12131 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:00.817004 12514 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:00.817358  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:00.682765 12353 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:00.823551  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:00.820030 12455 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:00.848349  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2548) took 1220 ms. Trace:
I20260709 13:03:00.822798 12487 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:03:00.840957 12507 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.665s	user 0.001s	sys 0.001s
W20260709 13:03:00.876331  8530 auto_rebalancer.cc:893] Removing replace marker failed: Replica d0a2b9c938d94a4fa258e068ea054d84 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:03:00.880872  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57399.8us
I20260709 13:03:00.848766 12353 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:00.852949  8651 rpcz_store.cc:276] 0709 13:02:59.627677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.627811 (+   134us) service_pool.cc:224] Handling call
0709 13:02:59.642892 (+ 15081us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:59.655979 (+ 13087us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:59.709670 (+ 53691us) log.cc:844] Serialized 14 byte log entry
0709 13:02:59.832411 (+122741us) log.cc:844] Serialized 14 byte log entry
0709 13:02:59.837035 (+  4624us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:59.850213 (+ 13178us) log.cc:844] Serialized 26 byte log entry
0709 13:02:59.850321 (+   108us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:59.850342 (+    21us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:59.850386 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:00.848028 (+997642us) raft_consensus.cc:1685] finished
0709 13:03:00.848057 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:00.848234 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5969,"prepare.queue_time_us":48070,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":60,"raft.queue_time_us":13612,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":18114,"spinlock_wait_cycles":13859840,"thread_start_us":68013,"threads_started":1,"wal-append.queue_time_us":116741}
W20260709 13:03:00.890344  8530 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 13:03:00.893580 12515 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 13:03:00.897732 12342 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:00.898974 12342 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.899206 12447 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:03:00.899668 12342 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:00.900254 12467 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:00.901767 12467 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:00.903473 12269 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:00.881986 12521 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:00.904526 12521 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:00.904889 12521 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:03:00.890782  8530 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 13:03:00.886603 12455 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:00.908766 12455 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:00.934563  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43594.7us
I20260709 13:03:00.853026 12514 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
W20260709 13:03:00.955686 12358 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.873s	user 0.008s	sys 0.003s
I20260709 13:03:00.941097 12329 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:00.964124 12342 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:00.964416 12312 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.195s	user 0.012s	sys 0.000s
I20260709 13:03:00.899791 12447 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:00.968514 12455 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:00.972728 12485 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:00.978359  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 692) took 1072 ms. Trace:
I20260709 13:03:00.983273  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2434) took 1517 ms. Trace:
I20260709 13:03:00.827703 12486 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:00.988636  8731 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.912s	user 0.009s	sys 0.000s
I20260709 13:03:01.012974  8728 rpcz_store.cc:276] 0709 13:02:59.461833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.464468 (+  2635us) service_pool.cc:224] Handling call
0709 13:02:59.464838 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:00.183719 (+718881us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.979312 (+795593us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54004722d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:00.979498 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1455959296}
I20260709 13:03:01.016561  8731 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:01.011999  9329 rpcz_store.cc:276] 0709 13:02:59.905904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.906054 (+   150us) service_pool.cc:224] Handling call
0709 13:03:00.978113 (+1072059us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b5400519354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:00.978267 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":270864640}
I20260709 13:03:01.018283  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2114) took 1036 ms. Trace:
W20260709 13:03:00.990284  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 356077us
I20260709 13:03:00.993705  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:00.994277  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.018847  8731 rpcz_store.cc:276] 0709 13:02:59.981668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.020779 (+ 39111us) service_pool.cc:224] Handling call
0709 13:03:01.018094 (+997315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:01.027520  8651 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:01.009173 12447 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:03:01.033061 12519 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 13:03:01.010389 12516 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.109s	user 0.000s	sys 0.006s
I20260709 13:03:01.038851  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 741) took 1302 ms. Trace:
I20260709 13:03:01.039892  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:01.048506  8699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.005s	sys 0.013s
W20260709 13:03:01.049146  8699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.005s	sys 0.013s
I20260709 13:03:01.039500  8576 rpcz_store.cc:276] 0709 13:02:59.736139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.736288 (+   149us) service_pool.cc:224] Handling call
0709 13:02:59.736617 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:59.754500 (+ 17883us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:59.754547 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:59.779380 (+ 24833us) log.cc:844] Serialized 26 byte log entry
0709 13:02:59.989263 (+209883us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:59.989319 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:59.989394 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.038566 (+1049172us) raft_consensus.cc:1685] finished
0709 13:03:01.038596 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.038770 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2890,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7371,"thread_start_us":149547,"threads_started":1,"wal-append.queue_time_us":258407}
W20260709 13:03:01.050845  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 233541us
W20260709 13:03:01.051671  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:01.052121  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 287009us
W20260709 13:03:01.052752  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:01.052750  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59171.9us
I20260709 13:03:01.053167  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:01.054198  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.056382  8731 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:01.040859 12260 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.059130  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:01.009070 12358 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.046878 12455 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:03:01.011128 12312 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.063222 12329 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:01.069169  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.069171  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:01.070515  8728 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:01.073614  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:01.075001  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 583) took 1070 ms. Trace:
I20260709 13:03:01.075398  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.040324 12526 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:03:01.075451  8650 rpcz_store.cc:276] 0709 13:03:00.004758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.004881 (+   123us) service_pool.cc:224] Handling call
0709 13:03:00.021360 (+ 16479us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:00.028460 (+  7100us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:00.028490 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:00.080513 (+ 52023us) log.cc:844] Serialized 52 byte log entry
0709 13:03:00.779712 (+699199us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:01.043350 (+263638us) log.cc:844] Serialized 14 byte log entry
0709 13:03:01.059314 (+ 15964us) log.cc:844] Serialized 14 byte log entry
0709 13:03:01.074678 (+ 15364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.074754 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.074792 (+    38us) raft_consensus.cc:1685] finished
0709 13:03:01.074806 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.074942 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32085,"prepare.queue_time_us":290521,"raft.queue_time_us":16980,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":12617,"thread_start_us":920388,"threads_started":2,"wal-append.queue_time_us":775432}
I20260709 13:03:01.072368 12468 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:01.069813 12329 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:01.065652 12496 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 13:03:01.106843  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:01.113171  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44058.5us
I20260709 13:03:01.111675  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:01.116493  9326 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:01.115727  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:01.098307 12526 consensus_queue.cc:1243] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:03:01.125655  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50328.2us
I20260709 13:03:01.127444  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:01.141359 12541 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.142351 12525 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.105578 12329 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:01.143365  8657 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:01.129969  9329 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:01.067792 12260 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:01.140774 12534 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:01.141978 12541 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.147028  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:01.146134  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:01.147660  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:01.148120  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:01.145195 12484 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.977s	user 0.009s	sys 0.000s
I20260709 13:03:01.108789 12533 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:01.108844 12530 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:01.150274  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 81160.2us
I20260709 13:03:01.108494 12528 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:01.109170 12467 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:01.150200  8651 raft_consensus.cc:2368] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9.
I20260709 13:03:01.112533 12342 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:01.153882  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:01.108803 12531 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:01.155403  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:01.159437  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:01.163623  9326 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:01.156527  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:01.130630 12487 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:01.166347  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.167701  8650 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:01.169601 12520 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 13:03:01.170971 12527 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:01.172317 12447 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.173374 12260 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:01.177420 12497 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.181121 12358 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:01.191871 12487 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:01.197119 12468 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:03:01.198483  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65963.3us
I20260709 13:03:01.200133  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.139114 12492 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:03:01.167550 12531 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.187136 12484 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.201396 12260 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:01.210866 12531 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:01.214789 12468 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:01.231678  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:01.232633  8733 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:01.217044 12533 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:03:01.216768 12342 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:01.223282 12497 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:01.236986  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 83188us
I20260709 13:03:01.245966  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2122) took 1160 ms. Trace:
I20260709 13:03:01.158816 12530 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.193660 12487 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.239130 12447 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:01.240792 12484 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.246562  8725 rpcz_store.cc:276] 0709 13:03:00.085071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.604951 (+519880us) service_pool.cc:224] Handling call
0709 13:03:00.605516 (+   565us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:01.006764 (+401248us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:01.006818 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:01.007464 (+   646us) log.cc:844] Serialized 26 byte log entry
0709 13:03:01.150676 (+143212us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:01.150719 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.150915 (+   196us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.245541 (+ 94626us) raft_consensus.cc:1685] finished
0709 13:03:01.245581 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.245845 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":170574,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":6273,"thread_start_us":384424,"threads_started":2,"wal-append.queue_time_us":197916}
I20260709 13:03:01.232649 12466 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:01.259116 12543 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 13:03:01.265375 12533 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:01.271073 12530 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:01.271875  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 43555.1us
I20260709 13:03:01.272742  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.255581  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:01.280048  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 124702us
I20260709 13:03:01.282716  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:01.293804  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2440) took 1517 ms. Trace:
I20260709 13:03:01.284822  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.297412  8729 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:01.294790  8727 rpcz_store.cc:276] 0709 13:02:59.776095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.890708 (+114613us) service_pool.cc:224] Handling call
0709 13:03:01.293478 (+1402770us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b54000893d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:01.293713 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41489664}
I20260709 13:03:01.274269 12466 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:01.304171  9329 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:01.285156 12530 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:01.294498 12533 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:01.305263  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:01.300973 12447 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:01.255671 12329 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.310741 12532 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:01.301363 12228 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 13:03:01.312964  8697 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.317654 12541 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:01.320125  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:01.321130  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.321288  9324 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:01.322621  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:01.323771 12468 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.324093 12358 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:01.324669 12358 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:01.324316 12468 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.325608 12358 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.326889 12468 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:01.329517  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:01.304802 12484 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:01.330220  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.336684  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:01.339391  9328 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:01.341022  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:01.324970  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 584) took 1319 ms. Trace:
I20260709 13:03:01.332326  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.343945  8658 rpcz_store.cc:276] 0709 13:03:00.005167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.024295 (+ 19128us) service_pool.cc:224] Handling call
0709 13:03:00.024673 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:00.763809 (+739136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.324683 (+560874us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:01.324902 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1516199168}
I20260709 13:03:01.341639  9328 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:01.348222  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:01.350806  9331 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:01.312116 12533 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.341740  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:01.354451 12549 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:03:01.355554 12528 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:01.354909  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:01.354619 12510 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:01.359582  8697 leader_election.cc:400] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:03:01.362715  8697 leader_election.cc:403] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 13:03:01.364384 12353 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:03:01.364971 12353 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:01.366127 12353 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.367831 12468 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:01.330636 12530 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.317703 12329 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:01.372534  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 218286us
I20260709 13:03:01.373787  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:01.374169  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 922) took 1304 ms. Trace:
I20260709 13:03:01.374578  9323 rpcz_store.cc:276] 0709 13:03:00.065625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.065759 (+   134us) service_pool.cc:224] Handling call
0709 13:03:00.066066 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:00.163962 (+ 97896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:00.163995 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:00.170179 (+  6184us) log.cc:844] Serialized 26 byte log entry
0709 13:03:00.786403 (+616224us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:00.786455 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.357459 (+571004us) spinlock_profiling.cc:243] Waited 66.7 ms on lock 0x7b5400386b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:01.357504 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.357549 (+    45us) raft_consensus.cc:1685] finished
0709 13:03:01.357575 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.369714 (+ 12139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30734,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":5233,"spinlock_wait_cycles":153366272,"thread_start_us":362694,"threads_started":1,"wal-append.queue_time_us":688950}
I20260709 13:03:01.353392 12533 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:01.369537 12486 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.381815  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:01.359036 12290 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.362694 12549 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:01.385162  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:01.350734 12260 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 13:03:01.313014 12342 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.393326  8654 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:01.352519 12497 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:03:01.392068 12518 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:01.392127 12549 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:01.312770 12466 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:01.409262  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 54435.2us
W20260709 13:03:01.412250  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 70606.6us
I20260709 13:03:01.406894  9297 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.413579  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:01.418558 12555 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:01.418994 12555 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.419332 12555 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:01.419730  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:01.422246  8725 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:03:01.422551  8731 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:01.423707  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:01.423993  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:01.425331  8731 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:01.426482 12486 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:01.427325 12486 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.429142  8731 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:01.434252 12486 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:01.434593  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:01.435299  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:01.435595  8657 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:03:01.436568  8725 raft_consensus.cc:2410] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:03:01.438184  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:01.440550  8731 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:01.442560  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:01.503892 12497 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.525997 12503 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:01.471479  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:01.477578 12471 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 5. Current term is 6.
I20260709 13:03:01.511344 12466 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:01.447258 12520 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:03:01.523584 12521 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.003s
I20260709 13:03:01.473883  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:01.452508  9331 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:03:01.513522  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 106740us
I20260709 13:03:01.393759 12290 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.404320 12228 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:01.404793 12518 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:01.450182  9328 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:01.414440 12447 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:03:01.528014 12554 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 13:03:01.471311 12558 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 13:03:01.544613  8727 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:03:01.552179  8725 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:01.552376  8732 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:01.551373 12531 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.564591 12531 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:01.541793 12503 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:01.568197  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 119435us
I20260709 13:03:00.565135  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:01.545131 12520 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:01.585029  8726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2119) took 1501 ms (client timeout 1614 ms). Trace:
W20260709 13:03:01.585743  8726 rpcz_store.cc:269] 0709 13:03:00.083944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.097308 (+ 13364us) service_pool.cc:224] Handling call
0709 13:03:01.584714 (+1487406us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400436554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:01.584939 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2722714368}
I20260709 13:03:01.591823 12549 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.593859 12228 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:01.591904  8726 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20260709 13:03:01.596199  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.594974 12228 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.595566 12549 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:01.606066  8656 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:03:01.599013  8731 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:03:01.714587  8724 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 13:03:01.715101  8724 pending_rounds.cc:85] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 7
I20260709 13:03:01.715335  8724 pending_rounds.cc:107] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 13:03:01.715696  8724 raft_consensus.cc:2889] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:01.549947 12521 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.718329  9330 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:01.719511 12521 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:01.720208 12518 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.720153 12520 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.447068 12555 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.597285 12528 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:01.721690 12518 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:01.589711 12447 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.175s
W20260709 13:03:01.799767  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 231127us
W20260709 13:03:01.803783  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 390233us
W20260709 13:03:01.803036  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 257719us
I20260709 13:03:01.806106  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:01.806908  8584 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:01.798573 12465 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.809502  8578 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
W20260709 13:03:01.808537  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.810760  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:01.814034  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:01.816386  8575 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:01.809513  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:01.802078 12549 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:01.822888  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:01.823037 12549 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.825141  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:01.825754 12572 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:01.821946  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.827363 12549 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:01.828404 12549 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:03:01.828749 12577 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:01.828918 12549 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:01.829845  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:01.830019 12549 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.829828  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:01.831465  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.832145 12533 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.807790 12468 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.834784  8652 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:01.839859  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:01.840271  9328 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:03:01.840790  9329 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:01.813050 12465 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:01.840667  9328 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:01.813297 12520 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:01.842820  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.844938 12358 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 13:03:01.845036  9328 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:01.845880  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:01.850497 12569 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 13:03:01.850862 12525 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:01.721097 12503 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.809916 12528 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:03:01.855593  9328 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:01.861547 12324 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:01.863442  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.880833 12290 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:01.843935 12468 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.888342  9328 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 13:03:01.891526  9328 pending_rounds.cc:85] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 6
I20260709 13:03:01.891973  9328 pending_rounds.cc:107] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 13:03:01.892390  9328 raft_consensus.cc:2889] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:01.884459 12260 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:01.895031 12260 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:01.895622 12533 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:01.898294 12468 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:01.900138 12555 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:01.903424  8658 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:01.905957  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.870656 12579 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:03:01.896534 12260 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.908938 12260 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:03:01.915553 12578 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.064s	user 0.002s	sys 0.000s
I20260709 13:03:01.916955 12579 consensus_queue.cc:1243] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:01.920361 12465 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.922935 12465 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:01.926075 12269 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:01.926602 12269 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:01.927368 12485 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:01.930191 12269 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.931252 12269 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:01.945581 12555 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:03:01.946671  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44290.9us
I20260709 13:03:01.947700 12555 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:03:01.948446 12512 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.228s	user 0.008s	sys 0.000s
I20260709 13:03:01.949168  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:01.949853 12175 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 13:03:01.950135 12555 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:03:01.951539  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2110) took 2026 ms (client timeout 2007 ms). Trace:
W20260709 13:03:01.952122  8733 rpcz_store.cc:269] 0709 13:02:59.925002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.934650 (+  9648us) service_pool.cc:224] Handling call
0709 13:03:01.951280 (+2016630us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400339854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:01.951434 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2702548352}
I20260709 13:03:01.952350  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:01.953660 12555 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:03:01.955541 12540 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.734s	user 0.002s	sys 0.000s
I20260709 13:03:01.957170  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:01.958415 12555 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:01.958953 12555 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:01.959419 12555 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:01.948566  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.964953  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:01.966615  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:01.967402  8733 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:01.968557  8724 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
W20260709 13:03:02.032111  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.053997  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.062466  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:02.063825  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.064576  8728 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:02.064159 12555 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:01.974967 12527 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.070452  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2536) took 2680 ms. Trace:
I20260709 13:03:02.072417  8582 rpcz_store.cc:276] 0709 13:02:59.386764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.387454 (+   690us) service_pool.cc:224] Handling call
0709 13:02:59.398575 (+ 11121us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:59.856866 (+458291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.067191 (+2210325us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54000b6154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:02.067469 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1052003200}
W20260709 13:03:02.073625  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.077052  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.082487  8733 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:02.071353 12579 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:03:02.098889  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 120169us
I20260709 13:03:02.100025  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:02.102653  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.105671  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.106570  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:02.074538 12527 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.101434 12555 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:02.106253 12528 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:02.108749 12572 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.118682  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.122807  8728 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 13:03:02.108636 12549 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 13:03:02.117933 12528 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.119186 12579 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:02.132606  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:02.131611 12528 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:02.142341 12527 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:02.146035  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2108) took 2221 ms. Trace:
I20260709 13:03:02.146452  8729 rpcz_store.cc:276] 0709 13:02:59.924472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.943459 (+ 18987us) service_pool.cc:224] Handling call
0709 13:03:01.294119 (+1350660us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.300364 (+  6245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.145787 (+845423us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000893d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:02.145944 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3105000320}
I20260709 13:03:02.137523 12573 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:02.154515 12573 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:02.168725  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:02.169297  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.167636  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:02.170068  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 776073us
I20260709 13:03:02.175107  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:02.164291 12573 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.194075  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.195814  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:02.169673 12526 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:02.200874  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.200698  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:02.201110  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:02.179955 12132 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.229116  8732 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:02.229856  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:02.177999 12260 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:03:02.229521  8733 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:02.232327  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.180341 12353 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.180531 12479 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.239903  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.197525 12514 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:02.210312 12468 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:02.181834 12171 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 13:03:02.199059 12548 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.198874 12518 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 13:03:02.244401 12268 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:02.244578 12447 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.251986  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.240613 12549 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:02.238027 12465 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.241974 12455 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:03:02.259765 12268 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.250993 12312 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:02.243836 12260 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:02.241865 12555 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:03:02.268325  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 70156.6us
I20260709 13:03:02.244917 12468 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:02.270393  8658 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:02.280166  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.280210  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.228896 12163 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 13:03:02.264003 12268 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:02.236965 12132 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.286180  8654 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:03:02.265982 12585 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 13:03:02.290246  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.288714  8651 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260709 13:03:02.288563  8650 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:02.290930  8651 pending_rounds.cc:85] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 5
I20260709 13:03:02.291265  8651 pending_rounds.cc:107] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260709 13:03:02.291643  8651 raft_consensus.cc:2889] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:02.239791 12479 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.241114 12353 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.291517  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.296267  8657 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:02.296823  8652 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:03:02.262907 12269 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:02.262799 12544 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.298496 12269 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.298678 12544 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.262678 12290 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:02.300112  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:02.299937 12290 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.279137 12228 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:02.298753 12581 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:02.302626  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:02.278812 12465 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.303783 12547 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:02.304355 12572 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:02.305642 12465 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:02.260918 12548 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.306689  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:02.307343  8726 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.283047 12468 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.310431  8724 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 13:03:02.310848  8724 pending_rounds.cc:85] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 7
I20260709 13:03:02.311088 12467 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:02.311458  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:02.260382 12598 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.312412 12598 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.315078 12290 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:02.316982 12598 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:02.317667 12577 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:02.243453 12525 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 13:03:02.319015  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2564) took 1312 ms. Trace:
I20260709 13:03:02.319605  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:02.320288  9323 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:02.311112  8724 pending_rounds.cc:107] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 13:03:02.321776  8724 raft_consensus.cc:2889] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:02.322057  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:02.322091  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 587) took 2315 ms. Trace:
I20260709 13:03:02.322700  9323 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.322309  8548 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:02.322922  8653 rpcz_store.cc:276] 0709 13:03:00.007135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.544762 (+537627us) service_pool.cc:224] Handling call
0709 13:03:00.545269 (+   507us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:00.951322 (+406053us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:01.067706 (+116384us) log.cc:844] Serialized 14 byte log entry
0709 13:03:01.540430 (+472724us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b1800472300. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a
0709 13:03:01.954649 (+414219us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:02.208840 (+254191us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:02.209514 (+   674us) log.cc:844] Serialized 78 byte log entry
0709 13:03:02.209694 (+   180us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:02.309786 (+100092us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b1800472300. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:03:02.309981 (+   195us) log.cc:844] Serialized 14 byte log entry
0709 13:03:02.310364 (+   383us) log.cc:844] Serialized 14 byte log entry
0709 13:03:02.310516 (+   152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.310588 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:02.321876 (+ 11288us) raft_consensus.cc:1685] finished
0709 13:03:02.321895 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:02.322039 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":637672,"prepare.queue_time_us":281101,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":1407,"raft.queue_time_us":998683,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":14721,"spinlock_wait_cycles":49913856,"thread_start_us":563760,"threads_started":4,"wal-append.queue_time_us":107575}
I20260709 13:03:02.272547 12171 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:02.272095 12503 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.319439  8656 rpcz_store.cc:276] 0709 13:03:01.006826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.006983 (+   157us) service_pool.cc:224] Handling call
0709 13:03:01.027136 (+ 20153us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.708101 (+680965us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.318752 (+610651us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b540005a0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:02.318942 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1330191872}
I20260709 13:03:02.336670  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:02.340672  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44515.1us
I20260709 13:03:02.355304  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:02.359057  9330 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.360179  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:02.363920  9328 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
W20260709 13:03:02.372207  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50589.1us
I20260709 13:03:02.382447 12539 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:02.382336  9323 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:02.382979 12539 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:02.383911 12539 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:02.384361  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 63559.5us
I20260709 13:03:02.384976 12539 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.385468 12447 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:02.386385 12465 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:02.335630 12290 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:02.394009  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:02.361603 12485 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:02.389276  9330 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:02.361984 12455 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:02.394420  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:02.361276 12520 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:03:02.395574  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 59007.5us
I20260709 13:03:02.368501 12349 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:02.319016 12468 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.427392 12468 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:02.356166 12324 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:03:02.404464 12447 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:02.437963 12587 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260709 13:03:02.407970 12581 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 13:03:02.390933 12549 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:02.408306 12547 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:02.439401 12549 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.439949 12549 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.364955 12497 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:03:02.444010 12132 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:02.361534 12171 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.444352 12497 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:02.444813 12497 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:02.396333 12598 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 13:03:02.445683 12497 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.422961  8576 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:02.446803 12353 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:02.447611 12592 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:02.448149 12592 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:02.448701 12543 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:02.410177 12455 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.450011 12455 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.445705 12548 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:02.446856 12497 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.451335 12324 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:02.454351 12269 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:02.454684 12163 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:02.455624 12163 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:02.456481 12518 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:02.456897 12518 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:02.429100  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:02.422806 12520 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:02.425122  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.405148 12485 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:02.423175  8726 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:02.458668  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:02.463286  9323 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:02.458340  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:02.429070  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:02.471896 12518 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.472571  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:02.472772  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:02.512778  9324 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:02.449048 12592 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:02.481830  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 56782us
I20260709 13:03:02.482770  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:02.489545  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:02.510208  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:02.560843  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 71404.7us
I20260709 13:03:02.408532 12290 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.463656 12542 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.391101 12260 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.410095 12546 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.099s
I20260709 13:03:02.485059 12485 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.508347 12268 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 13:03:02.569882 12593 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:03:02.572542  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62412.5us
I20260709 13:03:02.720319  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:02.573238 12342 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:03:02.574743  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 145714us
I20260709 13:03:02.711565  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:02.721148  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.487306 12548 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 13:03:02.548827 12521 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:02.765136  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:02.513706 12525 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.200s
I20260709 13:03:02.566067 12518 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.511582 12158 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.209s
W20260709 13:03:02.774525  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54277us
I20260709 13:03:02.669353 12447 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:02.718319 12353 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:02.764010 12342 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:02.797703  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 235947us
I20260709 13:03:02.769622 12465 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.794809 12256 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:02.719997 12132 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260709 13:03:02.787673 12324 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:02.805474 12604 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 13:03:02.809937  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:02.824215  8656 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.790889 12521 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:02.795753 12353 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.810282  8652 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:02.820454  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.831667  8653 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:02.714329 12520 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.724009 12542 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:02.802772 12342 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.841233  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.767897 12549 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.841972  8657 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
I20260709 13:03:02.816581 12268 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:02.808416 12479 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:02.823750 12492 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:02.845480 12590 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260709 13:03:02.823421 12256 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:02.875496  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:02.841071 12228 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:02.839577 12353 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:02.880965  8656 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:02.807695 12465 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.881873  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.889262  8652 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:02.889242  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:02.849426 12594 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:02.830113 12324 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:02.881767 12603 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:03:02.874560 12268 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:02.889829 12581 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:02.892063 12158 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:02.898428 12616 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:02.878129 12479 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:03:02.878428 12471 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:03:02.884565 12517 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.378s
I20260709 13:03:02.904857 12527 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:02.882390 12492 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:02.926422  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 161323us
I20260709 13:03:02.908332 12171 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:02.917940 12158 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.909520 12544 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:02.946724  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:02.932068 12601 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.377s	user 0.000s	sys 0.001s
I20260709 13:03:02.952744  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:02.953171  8732 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:03:02.935887 12471 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:02.923954 12616 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.960846  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.961841  8652 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:03:02.962783  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:02.963361  8652 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:03:02.969983 12614 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:03:02.970395 12383 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:02.970798 12383 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:02.971698  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.971628 12383 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.972359  8654 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:02.972683 12383 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.973348  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:02.974097  8654 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:02.974797  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.975834  9323 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:02.977703  9329 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:03:02.905756 12594 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.941166 12158 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.937191 12527 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:02.988895  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.989768  8731 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:03:02.986217  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.987882  9325 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:02.993754  8726 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:02.951622 12171 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:02.956003 12290 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:02.985384 12602 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
W20260709 13:03:02.985765 12605 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
I20260709 13:03:02.999506  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:02.922537 12521 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:02.959434 12471 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:03.001967  8733 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:02.954902 12544 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 13:03:02.998664  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:03.015053  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.020870  8731 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:03:02.992449 12595 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:03.017151  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.940623 12455 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.016544  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:03.026614  8725 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:03:03.042608  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:02.950701 12256 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.044929 12465 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.046752  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:03.044111 12609 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 13:03:03.012343 12171 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.055683  8654 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:03.041730 12479 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:03.041551 12549 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.059145  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2570) took 1685 ms. Trace:
I20260709 13:03:03.048723 12543 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:03.049448 12620 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:03.066493  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 177295us
I20260709 13:03:03.011816 12268 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.064807  8655 rpcz_store.cc:276] 0709 13:03:01.374086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.374203 (+   117us) service_pool.cc:224] Handling call
0709 13:03:01.382152 (+  7949us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:02.168199 (+786047us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:02.168231 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:02.171700 (+  3469us) log.cc:844] Serialized 26 byte log entry
0709 13:03:02.381672 (+209972us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:02.381711 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.381792 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:03.058834 (+677042us) raft_consensus.cc:1685] finished
0709 13:03:03.058863 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:03.059067 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":826039,"raft.queue_time_us":788256,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":22123,"thread_start_us":153436,"threads_started":1,"wal-append.queue_time_us":317327}
I20260709 13:03:03.051344 12542 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.081363  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:03.083202  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:03.079697  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63270.8us
I20260709 13:03:03.086362  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:03.040588 12595 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.089015  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:03.074476 12335 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:03.070828 12543 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:03.129077  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.138111  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.138777  8654 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
W20260709 13:03:03.143159  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 581743us
I20260709 13:03:03.143930  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.099669 12623 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:03.137496 12543 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:03.125319 12324 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.074985 12479 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:03.151265  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64991.2us
I20260709 13:03:03.153076  8652 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:03.063011 12471 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.155242 12633 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:03:03.156186 12618 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
I20260709 13:03:03.157243 12626 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:03.158649 12634 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 13:03:03.167752 12528 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.017s	user 0.013s	sys 0.000s
I20260709 13:03:03.066036 12527 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.163165  8549 leader_election.cc:400] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9.
I20260709 13:03:03.182314  8549 leader_election.cc:403] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:03.169044  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:03.170348  9329 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:03:03.175163  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.159965  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.184980  8659 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:03.089336 12620 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.165978 12594 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.171262 12324 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:03.188925  8659 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:03.173040 12626 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.191416 12633 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9.
W20260709 13:03:03.191862 12612 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 13:03:03.191409  8659 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:03.192394 12635 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:03:03.192619 12573 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.952s	user 0.008s	sys 0.004s
I20260709 13:03:03.177145 12471 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:03.182885 12616 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:03.207927  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53890.1us
I20260709 13:03:03.216619  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.181928 12528 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.221860  8582 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:03.222084 12633 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:03.222147 12532 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:03.222752 12532 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:03.223572 12532 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:03.224999 12627 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 13:03:03.221930  8577 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:03:03.240588  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.241371  8580 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:03.167480 12623 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.247133  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:03.250200  8580 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:03.247332  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.248318  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.252065  8575 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:03.199949 12324 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:03.249047  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.199515 12626 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.266196 12639 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.273885  8582 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:03.274106  8577 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:03:03.276815  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.282166 12620 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.283186  8579 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:03.284895  8579 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:03:03.285655  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:03.286325 12497 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.835s	user 0.009s	sys 0.002s
I20260709 13:03:03.287164  8583 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:03.288038  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:03.286919 12497 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.294049  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.294373  9324 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:03.294813  8583 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:03.298095  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 935) took 2002 ms. Trace:
I20260709 13:03:03.300246  8581 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:03.302100  8575 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:03:03.283852 12485 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.510s	user 0.010s	sys 0.000s
I20260709 13:03:03.305049  8580 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:03.312712  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.314267  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.314924  8582 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:03.274874 12549 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.326290  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.329859  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.330260  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.331805  8584 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:03.217306 12573 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.333422  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.330897  8575 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:03.242357 12158 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.331805  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:03.339084  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.342434  8580 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:03.299347  9326 rpcz_store.cc:276] 0709 13:03:01.296099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.296278 (+   179us) service_pool.cc:224] Handling call
0709 13:03:01.296620 (+   342us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:01.299065 (+  2445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:01.299101 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:01.299851 (+   750us) log.cc:844] Serialized 78 byte log entry
0709 13:03:01.392848 (+ 92997us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:01.547953 (+155105us) log.cc:844] Serialized 14 byte log entry
0709 13:03:02.303500 (+755547us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b48001b06d8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:03.227454 (+923954us) spinlock_profiling.cc:243] Waited 93.2 ms on lock 0x7b180035e5a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf
0709 13:03:03.227590 (+   136us) log.cc:844] Serialized 14 byte log entry
0709 13:03:03.297537 (+ 69947us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.297660 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:03.297727 (+    67us) raft_consensus.cc:1685] finished
0709 13:03:03.297745 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:03.297949 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":631122,"prepare.queue_time_us":272106,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":101,"raft.queue_time_us":688141,"raft.run_cpu_time_us":2478,"raft.run_wall_time_us":65225,"spinlock_wait_cycles":217925248,"thread_start_us":695065,"threads_started":5,"wal-append.queue_time_us":160834,"wal-append.run_cpu_time_us":2724,"wal-append.run_wall_time_us":1149797}
I20260709 13:03:03.340981  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.345263  8584 raft_consensus.cc:2410] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:03:03.348333  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:03.349318  8579 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:03.348695  8582 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:03.357358  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:03.336010 12503 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.974s	user 0.009s	sys 0.003s
I20260709 13:03:03.358528  8584 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:03:03.372653  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.375509  8576 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:03.378472  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.298615 12497 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.378538  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.386301  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.387012  8583 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:03.379985  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.336098 12577 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:03.393616  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 117928us
I20260709 13:03:03.381982  8578 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 13:03:03.394677  8578 pending_rounds.cc:85] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 5
I20260709 13:03:03.322077 12485 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.386067 12471 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.391381  8583 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:03:03.391714  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:03.394807  8580 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:03.395035  8578 pending_rounds.cc:107] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 13:03:03.405791  8578 raft_consensus.cc:2889] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:03.396631  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:03.390313  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.407136  8582 raft_consensus.cc:2376] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9. Re-sending same reply.
W20260709 13:03:03.381956 12636 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 13:03:03.414160  8575 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:03:03.335070 12549 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.421887  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.299957 12639 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.422797  8656 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.426369  8583 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:03.383865 12595 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:03.439567  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.446671  8579 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:03.452279  8577 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:03.455315  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.427129 12479 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:03.473342  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 81710.7us
I20260709 13:03:03.475497  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.409452 12577 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.424414 12471 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.479991  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:03.481204  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 136649us
W20260709 13:03:03.482797  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.484064  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.394083 12324 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.466531 12268 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:03.485289  8656 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:03.468809 12643 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:03.459329 12527 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 13:03:03.476099 12479 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.500411 12447 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:03.500944 12447 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:03.502987 12471 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.503702 12471 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:03.504578 12492 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.506552 12639 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:03.506945 12447 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.508437 12447 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:03.511596 12577 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.513813 12497 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.517958 12573 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.518327 12171 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.488770 12304 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:03:03.520980 12549 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.488457 12528 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.493947 12158 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:03.524277 12342 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.661s	user 0.009s	sys 0.003s
I20260709 13:03:03.525130 12527 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:03.526629 12479 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:03.528035 12518 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.745s	user 0.008s	sys 0.003s
I20260709 13:03:03.497751 12268 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:03.517202 12353 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.618s	user 0.008s	sys 0.001s
I20260709 13:03:03.533166 12655 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:03.534118  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.538955  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:03.557825  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66492.9us
I20260709 13:03:03.566473  8651 raft_consensus.cc:2376] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 10. Re-sending same reply.
I20260709 13:03:03.573434  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.574187  8656 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:03.552528 12527 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:03.577754  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.579427  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.581377  8651 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:03.582599  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.565315 12623 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.568181 12655 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:03.584493  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.584363  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.599248  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.564132 12158 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:03:03.602512  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.567233 12353 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.607435  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:03.563695 12342 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.608103  8651 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:03.585722 12480 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:03.582559 12485 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:03.614557  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.614692  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.617911  8656 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 6: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:03.577288 12643 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:03:03.583328 12304 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 13:03:03.596783 12655 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.582309 12518 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:03:03.585654 12468 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 13:03:03.612862 12480 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.601909 12623 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 13:03:03.620141 12620 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:03.640502 12620 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.640909 12620 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:03.643472 12527 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.646710 12353 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:03.611747 12171 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 13:03:03.648221 12268 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.654471 12643 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:03.654670 12342 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:03.655046 12643 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:03.653491 12620 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.633634 12594 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
W20260709 13:03:03.657799  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50446.8us
I20260709 13:03:03.658962  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.666719  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:03.678155  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:03.681027 12652 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:03:03.683259  8659 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:03.657368 12471 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 13:03:03.683621 12342 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:03.684760  8659 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:03:03.637773 12657 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 13:03:03.655926 12480 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:03.690001 12543 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.004s
I20260709 13:03:03.657366 12479 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:03.691006 12479 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:03.690056  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:03.693377  8659 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:03.698364  8733 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:03.700203 12447 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:03.701046 12447 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:03.701445 12447 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:03.701143  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.695655  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:03.703188  9326 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:03.659898 12633 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:03.694427  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:03.694499  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:03.735829  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:03.736704  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:03.739711  8725 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:03.637593 12592 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 13:03:03.708451  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:03.693447 12467 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
W20260709 13:03:03.754199  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 169877us
I20260709 13:03:03.757743  8581 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:03.741123  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46779.9us
W20260709 13:03:03.756428  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48012.8us
I20260709 13:03:03.758605  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:03.758757  8724 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:03:03.702894 12171 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:03.690471 12543 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.753437 12594 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:03.785778  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:03.787181  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:03.762602 12650 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:03.762784 12171 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:03.792104  9327 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:03.793023  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:03.793351  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:03.782492 12662 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 13:03:03.802649  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:03.706043 12447 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.792644 12171 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:03.803968  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:03.807941  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:03.811080 12447 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:03.806699  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.812848  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:03.791241 12650 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:03.816424  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.816951  9330 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
W20260709 13:03:03.817955 12660 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 13:03:03.818423  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.756577 12643 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:03.829888  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.830054  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.873015  8658 raft_consensus.cc:2376] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7. Re-sending same reply.
I20260709 13:03:03.874254  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.875036  8658 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:03:03.837731  9330 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:03.844272  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:03.790560 12633 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 13:03:03.782361 12655 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:03.814244 12626 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.514s	user 0.012s	sys 0.000s
I20260709 13:03:03.809517 12623 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:03:03.828596  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.849803  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.881732  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.878459  8575 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:03.832238 12480 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.884562  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.887984  8658 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:03:03.888106  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:03.895550  9325 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:03.889830 12670 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:03:03.845558 12650 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.847445 12617 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:03.899569 12672 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:03:03.845871 12581 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:03.902696  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.903285  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:03.915469  9326 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.904096  9330 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:03.903836  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.896273  8658 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:03.891034 12639 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.897150 12623 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.914403  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:03.919312  8729 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:03.935488  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:03.936236  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.937683  8658 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:03:03.939544  8651 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:03.944002  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.951208  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:03.915335 12580 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:03.940546 12623 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:03.909992 12592 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 13:03:03.896628 12626 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:03.946290 12304 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:03.972635  9325 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:03.914489 12664 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 13:03:03.932399 12617 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:03.989751  8658 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:03.932761 12579 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
W20260709 13:03:04.072784  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.096413 12324 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:04.122504  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.939167 12639 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.123767  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:03.976830 12304 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.939217 12665 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 13:03:04.127823  8658 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 6.
W20260709 13:03:04.135398  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 221034us
W20260709 13:03:04.163316  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 245588us
I20260709 13:03:04.166656  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:04.168754  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 221208us
I20260709 13:03:03.968989 12260 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.160s
I20260709 13:03:03.968407 12633 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.161s
I20260709 13:03:03.959604 12581 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.120466 12592 consensus_queue.cc:1243] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 13:03:04.174237  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.131723 12304 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.191191  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:04.177204  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:04.200393  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.131021 12543 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:04.173074 12579 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:04.213807  8658 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:04.128391 12324 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.011422 12580 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.206977 12260 consensus_queue.cc:1243] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:04.229298 12581 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.220885 12592 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:04.231297 12592 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:04.232301 12592 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.224814  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:04.234126  9328 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.235630 12580 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.237054  8655 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:03:04.218284 12480 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:04.227450 12324 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.227643  8580 raft_consensus.cc:2410] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:03:04.170956 12573 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 13:03:04.207388 12503 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:04.233423  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:04.234023 12675 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 13:03:04.245456  9328 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:04.238123  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:04.208608 12620 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.246253  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:04.250521 12655 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:04.255486 12626 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:04.207866 12171 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.258054 12171 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.251647 12503 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.246135 12680 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:04.271703 12680 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:04.272683 12503 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.272601 12680 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.273962 12680 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:04.274062 12617 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.251856  8724 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:04.277241  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.277878  8582 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:04.282299  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.283105  8659 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:04.285210  8658 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:04.285264  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:04.288362  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.289199  8576 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:04.299978  9323 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.003s	sys 0.004s
W20260709 13:03:04.302028  9323 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.004s	sys 0.005s
I20260709 13:03:04.300689  9326 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:04.308588  8582 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:04.310336  8582 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:04.314503  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:04.317787  8582 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:04.303748 12521 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:04.329104  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:04.340399  8724 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:04.348532  8576 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:04.331081  8582 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:04.360280  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 946) took 1575 ms. Trace:
I20260709 13:03:04.329233 12664 consensus_queue.cc:1243] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:03:04.360729  9324 rpcz_store.cc:276] 0709 13:03:02.785239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.788746 (+  3507us) service_pool.cc:224] Handling call
0709 13:03:02.789172 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:03.294797 (+505625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.360060 (+1065263us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54007dc3d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:04.360224 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147570048}
W20260709 13:03:04.365934  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.331664 12521 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:04.361109 12616 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.036s	user 0.009s	sys 0.019s
I20260709 13:03:04.367713 12650 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.370688 12521 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:04.378947  8727 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:04.379706 12650 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:04.380808 12479 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.689s	user 0.006s	sys 0.006s
I20260709 13:03:04.373013 12620 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
W20260709 13:03:04.382081  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.381295 12479 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:04.385937 12304 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:04.383354 12616 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.061s	user 0.009s	sys 0.021s
I20260709 13:03:04.389240 12531 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:04.389693 12531 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:04.389864 12304 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:04.390522 12531 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.392004 12531 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:03:04.403057  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.415495 12555 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:04.416282 12555 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:04.423445  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.424083  8658 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:04.424335  8548 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:04.426290  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.426041 12527 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.449046  9323 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:04.461229  9325 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:03:04.463538  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:04.464241  8576 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:03:04.465144  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:04.434602 12680 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.470252  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:04.435987 12691 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.464964  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:04.457484 12649 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.089s	user 0.013s	sys 0.007s
I20260709 13:03:04.473672  8582 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:04.436832 12549 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:04.461966 12503 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 13:03:04.486094  8724 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:04.436975 12693 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:03:04.472174  8622 connection.cc:256] RPC call timeout handler was delayed by -1.2026691129999563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3885s for a response
I20260709 13:03:04.491273  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.492067  8582 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.491581  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:04.494477  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:04.500412  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:04.503334  8650 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:04.492925 12549 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.507105 12549 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.507717  8576 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 5.
I20260709 13:03:04.508343 12171 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:04.509479 12573 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:04.493175 12620 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:04.510478 12555 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.470085 12617 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:03:04.510586 12620 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.511999 12592 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:04.512418  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:04.512760 12620 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.513068  8579 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:04.513499 12132 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:04.498971 12693 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:04.513903 12132 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:04.513991  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:04.514629  8579 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:03:04.514729 12693 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.515453  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.516996  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:04.481559 12649 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.110s	user 0.013s	sys 0.007s
I20260709 13:03:04.488898 12691 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.525494  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2567) took 1216 ms. Trace:
I20260709 13:03:04.521100  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:04.525882  8581 rpcz_store.cc:276] 0709 13:03:03.301916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.302931 (+  1015us) service_pool.cc:224] Handling call
0709 13:03:04.517818 (+1214887us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54002cb5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:04.517941 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":990235392}
I20260709 13:03:04.524276  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:04.524983  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:04.528205  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:04.507779 12696 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 13:03:04.534380  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:04.477618 12680 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.541848  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:04.544868  8582 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:04.546056  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:04.547075  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:04.543857 12521 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.525712 12503 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.531888 12592 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:04.531271 12256 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
I20260709 13:03:04.618284 12256 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:04.609895 12290 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:04.618709 12256 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:04.618898 12455 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.618928  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:04.619274 12256 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.548835 12649 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:04.619839 12649 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:04.619984  8657 raft_consensus.cc:2410] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:03:04.613590 12525 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:04.623021 12680 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:04.623539 12623 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.000s
I20260709 13:03:04.624053 12623 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.549389 12171 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:04.626477 12623 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:04.560114 12467 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:04.514770 12132 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.628738 12132 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:04.616153 12503 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.631086 12503 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.634874 12520 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.637244 12616 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.637496  8655 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:04.637853 12616 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.617125 12592 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:04.639012 12592 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.639490 12639 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.640098 12616 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:04.548156 12667 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:04.654603 12643 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:04.660696  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:04.663714  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 116671us
I20260709 13:03:04.664561  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:04.618753 12290 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:04.667754  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:04.661787  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:04.670060  9326 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:04.655234 12643 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.620730 12649 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.617332 12691 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:04.678639  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.679369 12667 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:04.681151  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:04.689636  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.694017  9328 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:04.694566  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:04.698632  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:04.698792  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.699755  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.699957  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.700414  8657 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:04.700894  8658 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:04.700980 12633 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:04.560315 12666 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
I20260709 13:03:04.695855  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:04.621291 12525 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.710994  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:04.717597  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.720023  8656 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:03:04.673761 12620 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:04.722419  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.730674  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.735036  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:04.738353  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.739867  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:04.742626  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:04.745607  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.745121  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:04.744089  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:04.701489 12633 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:04.743213  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:04.751891  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:04.753616  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:04.716050 12623 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:04.758473  9328 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:04.759027  9323 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:04.759737  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:04.759899  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:04.760262  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:04.760360  8575 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:04.702689 12335 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 13:03:04.744360 12354 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:04.744418 12471 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:03:04.789034  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44811.9us
I20260709 13:03:04.744483 12665 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.752010 12709 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:04.756783 12657 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:04.784976 12689 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:04.790594  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2443) took 1470 ms. Trace:
I20260709 13:03:04.753362 12594 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:04.790822  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:04.756457 12467 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:04.750059 12349 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:04.794095  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:04.769026  8579 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:04.753721 12531 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:04.757756 12479 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.795960 12626 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:03:04.800753  8470 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.310s	user 0.006s	sys 0.017s
I20260709 13:03:04.762331 12684 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.779151 12171 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:03:04.799875  8659 rpcz_store.cc:276] 0709 13:03:03.319740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.699159 (+379419us) service_pool.cc:224] Handling call
0709 13:03:03.699425 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:04.289980 (+590555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.786578 (+496598us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b5400131ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:04.790532 (+  3954us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1341601664}
I20260709 13:03:04.741556 12503 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 13:03:04.813587  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 600) took 1869 ms. Trace:
I20260709 13:03:04.814743  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2592) took 1975 ms. Trace:
I20260709 13:03:04.843966  8650 rpcz_store.cc:276] 0709 13:03:02.839005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.839307 (+   302us) service_pool.cc:224] Handling call
0709 13:03:02.865645 (+ 26338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:04.503761 (+1638116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.814458 (+310697us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400410854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:04.814681 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3765272960}
I20260709 13:03:04.815500 12486 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:04.816015 12471 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:04.817734  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.818473 12610 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:04.754400 12620 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 13:03:04.857896  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:04.742812 12711 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 13:03:04.823184  9326 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:04.820557  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2577) took 1018 ms. Trace:
W20260709 13:03:04.827378 12702 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260709 13:03:04.827243 12531 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.828341 12479 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.825047 12655 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:04.824837 12349 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.833043  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:04.885751  8579 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:04.887596  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:04.841977  8652 rpcz_store.cc:276] 0709 13:03:02.943769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.170899 (+227130us) service_pool.cc:224] Handling call
0709 13:03:03.171335 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.215149 (+ 43814us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:03.215177 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:03.247773 (+ 32596us) log.cc:844] Serialized 26 byte log entry
0709 13:03:03.399618 (+151845us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:03.399673 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.806153 (+1406480us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b5400352ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:04.806205 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:04.806260 (+    55us) raft_consensus.cc:1685] finished
0709 13:03:04.806273 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:04.813509 (+  7236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2241,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"spinlock_wait_cycles":98232576,"thread_start_us":78774,"threads_started":1,"wal-append.queue_time_us":161832,"wal-append.run_cpu_time_us":1961,"wal-append.run_wall_time_us":1055601}
I20260709 13:03:04.888491  8724 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:04.890702  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:04.891546  8730 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:04.855616  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:04.866748  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:04.829107 12626 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:04.893697 12626 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:04.860306  8578 rpcz_store.cc:276] 0709 13:03:03.801914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.858927 (+ 57013us) service_pool.cc:224] Handling call
0709 13:03:03.859349 (+   422us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:03.865821 (+  6472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:03.865849 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:03.882278 (+ 16429us) log.cc:844] Serialized 52 byte log entry
0709 13:03:04.314163 (+431885us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:04.335542 (+ 21379us) log.cc:844] Serialized 14 byte log entry
0709 13:03:04.819925 (+484383us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b1800305bc0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f14097ccd70 00007f14097c0293 00007f14097b5cea 00007f14097b623a 00007f1409c20a2a 00007f1409c300bf 00007f1409c30038
0709 13:03:04.820172 (+   247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.820249 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:04.820286 (+    37us) raft_consensus.cc:1685] finished
0709 13:03:04.820366 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:04.820498 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":457370,"prepare.queue_time_us":21066,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":48,"raft.queue_time_us":12157,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":7328,"spinlock_wait_cycles":23038208,"thread_start_us":413173,"threads_started":1,"wal-append.queue_time_us":426545}
I20260709 13:03:04.779505 12290 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.840853 12171 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:04.797712 12665 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.902849 12531 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.856670 12521 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.905264  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:04.906332 12655 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:04.867303 12610 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.909204 12710 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.848093 12689 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:04.848875 12486 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:04.911908  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:04.823982 12657 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:04.920205  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64707.9us
I20260709 13:03:04.951903  8732 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:04.952633  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:04.831560 12639 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 13:03:04.919188  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:04.960131  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:04.960628  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:04.904099 12704 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:03:04.927855 12349 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.928399 12335 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:04.917157  8730 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:04.847169 12684 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:04.932514  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 133533us
I20260709 13:03:04.964681  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:04.863541 12546 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.919005 12171 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:04.825927 12633 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.919467 12667 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 13:03:04.962499  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:04.935927  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 141888us
I20260709 13:03:04.921470 12467 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:04.938777 12531 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:04.971165  8657 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:04.972849  8730 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.974342  8652 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:05.128047  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:04.974493  8579 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:05.130079  8652 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:04.963037 12342 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:05.012010  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.001788  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.131666  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:05.135684  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 173244us
I20260709 13:03:05.136726  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:05.010365  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:05.138015  8728 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:04.990916 12665 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.958156 12521 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:04.958451 12594 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:04.964030 12643 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.964293 12663 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:04.981117 12712 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 13:03:04.980469 12711 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.147197  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:04.993259 12684 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:04.999081 12579 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:04.983933 12335 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:04.994758 12479 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.143705 12714 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:05.134423 12577 raft_consensus.cc:2764] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:03:04.981117 12610 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:04.999140 12657 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.154107  8730 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:04.983734 12525 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.142521 12485 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:05.155464 12711 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:05.156599 12674 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.984s	user 0.007s	sys 0.000s
I20260709 13:03:05.158946  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:05.159499 12705 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.320s	user 0.001s	sys 0.000s
I20260709 13:03:05.170035 12579 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:04.980893 12486 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.194866  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 718) took 1494 ms. Trace:
I20260709 13:03:05.195098 12486 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:04.963665 12471 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.199591 12711 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.127331 12555 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 13:03:04.990073 12256 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.142125 12527 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:05.134084 12531 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:05.167310 12626 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.201078 12555 consensus_queue.cc:1243] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:05.215847 12626 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:05.148689  8652 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.216346 12256 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:05.001935  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.217134 12527 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:05.217322  8652 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:05.218116 12527 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.219458  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:05.220785 12256 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:05.134615 12342 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:05.142777  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:05.221525 12531 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:05.221943  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 79248.5us
I20260709 13:03:05.222059 12175 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:05.222882  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:05.223104 12175 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:05.223901  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.224144 12175 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.993155 12710 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:05.145149 12521 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:04.980311 12689 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.207122 12467 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.169153 12663 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:04.982646 12655 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.178298  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.256918  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:05.239837  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.258069  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:05.159921  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 751) took 1369 ms (client timeout 1726 ms). Trace:
I20260709 13:03:05.258638  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:05.258984  8727 rpcz_store.cc:269] 0709 13:03:03.790175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.791562 (+  1387us) service_pool.cc:224] Handling call
0709 13:03:05.159759 (+1368197us) spinlock_profiling.cc:243] Waited 585 ms on lock 0x7b54001eafd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:05.159866 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1344514304}
I20260709 13:03:05.031446 12546 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:04.991690 12592 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.170686 12718 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:05.170529 12717 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:05.195462  9331 rpcz_store.cc:276] 0709 13:03:03.700536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.700658 (+   122us) service_pool.cc:224] Handling call
0709 13:03:03.700989 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.701382 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:03.701429 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:03.701933 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 13:03:03.912142 (+210209us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:03.912180 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.912254 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:05.155961 (+1243707us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:05.194468 (+ 38507us) raft_consensus.cc:1685] finished
0709 13:03:05.194500 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:05.194784 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3924,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":41763,"spinlock_wait_cycles":23746432,"thread_start_us":209589,"threads_started":1,"wal-append.queue_time_us":267638}
I20260709 13:03:05.206524  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:05.168718  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 216090us
I20260709 13:03:05.226079  8728 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:05.226297  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:05.250196  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.230957 12594 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:05.235471  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.970515 12715 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.206s
I20260709 13:03:05.230793 12455 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.254287  8652 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 13:03:05.308175  8652 pending_rounds.cc:85] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 5
I20260709 13:03:05.308480  8652 pending_rounds.cc:107] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 13:03:05.229315 12521 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.308771  8652 raft_consensus.cc:2889] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:05.265998  8730 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20260709 13:03:05.308617  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.235625  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.225358 12573 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.306496  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.312569  8732 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:05.310015  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:05.237015 12543 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:05.264130 12639 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.338936 12525 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
W20260709 13:03:05.344777  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.344409 12639 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.279502 12710 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:05.226894  8470 maintenance_manager.cc:643] P b2a6e2bdaa2043fba12c0ef60545b7fb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 8.653s	user 0.176s	sys 0.104s Metrics: {"bytes_written":29753,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":71499,"dirs.run_cpu_time_us":442,"dirs.run_wall_time_us":1211,"drs_written":1,"lbm_read_time_us":401,"lbm_reads_lt_1ms":4,"lbm_write_time_us":281001,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":5,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":25,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":228828,"threads_started":2,"update_count":11,"wal-append.queue_time_us":265495}
I20260709 13:03:05.337870  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:05.349206 12649 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:05.310814 12717 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:05.316490 12718 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:05.314857 12657 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:05.364696  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:05.372102 12468 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:05.374387  8724 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 9. Ops: []
I20260709 13:03:05.376088  8724 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:05.378095 12639 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:05.355764 12717 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:05.369215 12610 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:05.379040 12610 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.381908 12532 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:05.368878 12665 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:05.382910  8471 maintenance_manager.cc:419] P b2a6e2bdaa2043fba12c0ef60545b7fb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 8971 bytes on disk
I20260709 13:03:05.369004 12657 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.313745 12455 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.382763 12665 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:05.318068 12573 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.390974 12468 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:05.458992  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2565) took 2185 ms. Trace:
I20260709 13:03:05.459326 12468 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.337345 12342 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.448158 12718 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.446336  8470 maintenance_manager.cc:643] P b2a6e2bdaa2043fba12c0ef60545b7fb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.060s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":358,"lbm_reads_lt_1ms":4}
I20260709 13:03:05.414283  8724 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:05.385555 12610 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:05.463876  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 154035us
I20260709 13:03:05.483764  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:05.459446  8577 rpcz_store.cc:276] 0709 13:03:03.273881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.289365 (+ 15484us) service_pool.cc:224] Handling call
0709 13:03:05.458784 (+2169419us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54002cbad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:05.458923 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319067904}
I20260709 13:03:05.456671  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.486274  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:05.474210 12665 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.475247 12455 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.475461 12171 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.476691 12657 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.478135 12663 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:05.477833  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.462504 12721 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:05.475248 12573 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.477825  9331 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:03:05.469329  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.471050 12715 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:05.470132 12485 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.499464  8724 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:05.503285 12610 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.508612 12715 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.494994 12342 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:05.511546  9297 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.726s (SENT) [suppressed 20 similar messages]
I20260709 13:03:05.514611  8726 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:03:05.516556  8726 pending_rounds.cc:85] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:03:05.516836  8726 pending_rounds.cc:107] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:03:05.517113  8726 raft_consensus.cc:2889] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:05.510627 12455 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:05.515739 12715 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:05.531778 12737 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:05.515573 12610 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.540050 12610 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.541117 12620 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:05.541643 12738 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:05.541942 12620 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.542693 12592 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.510763 12721 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.544389 12620 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.513202 12485 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.509783 12735 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:05.547762 12485 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.548799  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:05.554900  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:05.555819  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65880us
I20260709 13:03:05.558342  8653 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:05.561728  8655 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:05.562248  8659 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: []
I20260709 13:03:05.563568  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.565423  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.545208 12721 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.516184 12543 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:05.573000  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 61493.6us
I20260709 13:03:05.573769  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.577042  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.577499  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.578274  9326 raft_consensus.cc:2410] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 13:03:05.579674  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.581110  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.580547  8727 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:05.562825 12455 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:05.585053 12455 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.561898 12573 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:05.587942 12573 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.588325 12573 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:05.592695  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.593441  8659 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 6.
I20260709 13:03:05.593948 12171 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.584223  9324 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:05.595454  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.596028  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.594731 12543 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.596640  8657 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:05.596161  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 940) took 1156 ms. Trace:
W20260709 13:03:05.598815  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.594643  8655 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:03:05.600387  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.598932  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.597656  9322 rpcz_store.cc:276] 0709 13:03:04.439786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.439982 (+   196us) service_pool.cc:224] Handling call
0709 13:03:05.595305 (+1155323us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:03:05.596108 (+   803us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4520832}
I20260709 13:03:05.601608  9324 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:05.603582  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.603648  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.608403  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.573168 12175 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.610076  9326 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:05.618969 12717 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.622395  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.623137  8657 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:05.625244  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:05.627310  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57619.4us
I20260709 13:03:05.571890 12485 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 13:03:05.629756  8727 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:05.638913  8655 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:05.658910 12573 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.670859 12573 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.667572 12715 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:05.673141  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62837.4us
I20260709 13:03:05.674086 12715 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.674819  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:05.675429 12748 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:05.676496 12748 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.675463  8655 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:05.679683  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:05.682053 12748 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.682960  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.683612  8583 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:05.684017  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.684592  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.684762  8584 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:05.685151  8579 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:05.687191  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.678120  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:05.688894  8728 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:05.599929 12521 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.621225 12175 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.697952  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.698166  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.705156  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.705150  8575 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:05.707803  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:05.708688  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.709619  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.709666  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.710400  8581 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:05.708788  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:05.716686  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.718904  8655 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:05.719719  8579 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:05.724195  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.725023  8581 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:05.725905 12717 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.726480  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.728102  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.728734  8581 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:05.690133 12171 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:05.723150  8579 raft_consensus.cc:1217] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Deduplicated request from leader. Original: 3.3->[4.4-10.5]   Dedup: 4.4->[10.5-10.5]
I20260709 13:03:05.695036 12755 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.779678  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:05.729538 12349 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:05.782498 12175 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.781706  8583 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:05.751886  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.784981  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:05.818347  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:05.819082  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.787557  8728 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:05.819304  8658 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:05.819196  8730 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:05.729364 12715 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:05.822263  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.803287  8579 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [4.4-10.5]
I20260709 13:03:05.693950 12531 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.830854  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.708010 12610 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.094s
I20260709 13:03:05.838446  8658 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:05.839623  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.840270  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.841455  8658 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:05.831916  8579 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:03:05.821447  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:05.818671  9323 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:05.844401  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.845487  8579 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:03:05.833199 12655 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:05.882289  8582 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:05.853173  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:05.827728 12717 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.891708  8659 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:05.830924 12715 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.893926  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.919831  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.922717  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.818310 12349 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.883617 12532 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:05.931216  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.926641  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 77000us
I20260709 13:03:05.846972 12721 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.925800  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.933792  8579 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:05.818549 12171 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.846895 12175 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:05.934824  8575 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 13:03:05.914955 12709 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.066s	user 0.030s	sys 0.029s
I20260709 13:03:05.915057 12555 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:05.885437 12518 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:05.923947 12715 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:05.929014 12290 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:05.931517 12349 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:05.940227  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:05.937047 12709 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.088s	user 0.031s	sys 0.029s
I20260709 13:03:05.936115 12175 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.936610 12532 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.946820 12518 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.944500 12555 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.916057 12655 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:05.938553 12594 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:05.828096 12755 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:05.950709  8575 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:05.956970  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.982511  8582 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:05.957731  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:05.965598  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:05.975134  8579 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:05.985157  8654 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.975198  8584 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:05.963449  8580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2574) took 2162 ms (client timeout 1886 ms). Trace:
I20260709 13:03:05.980294  8583 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:05.994206  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.994902  8656 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
W20260709 13:03:05.986714  8580 rpcz_store.cc:269] 0709 13:03:03.801031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.803661 (+  2630us) service_pool.cc:224] Handling call
0709 13:03:05.963251 (+2159590us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b5400320fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:05.963377 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":970180992}
I20260709 13:03:05.999157  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:05.999462  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:05.943702 12663 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.001215  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:05.927577 12717 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:05.951412 12268 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
W20260709 13:03:05.966323 12733 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.585s	user 0.003s	sys 0.000s
W20260709 13:03:05.969937 12750 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 13:03:06.003798  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:05.962607 12531 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:05.959297 12649 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.610s	user 0.008s	sys 0.001s
I20260709 13:03:06.006333  9329 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:05.980469 12754 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.308s	user 0.002s	sys 0.000s
I20260709 13:03:05.949741 12709 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:05.967911 12518 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.954068 12175 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:05.970732 12555 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.957445 12532 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.972692 12655 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:05.953495 12171 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:05.959830 12521 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:05.980064 12755 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.011423  8575 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:05.974313 12747 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 13:03:06.020771  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:06.033095  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:06.047217  9324 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:05.977816 12594 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:06.033433  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:06.054286  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:06.038651  8654 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
W20260709 13:03:06.026255 12732 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.648s	user 0.000s	sys 0.003s
I20260709 13:03:05.940644 12721 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:06.033548  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.061746  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:06.065752 12171 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:06.066668 12171 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.074426  8654 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:06.077843 12649 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:06.078549 12649 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:06.079042 12649 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:06.083093 12594 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.008244 12663 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:06.100252  8654 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:06.101176  8580 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:03:06.091914 12514 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:06.106309 12514 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.112399  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:06.115880  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:06.135200 12739 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:06.142680 12663 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:06.143582  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:06.098925 12175 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.146117  8580 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:06.146647 12175 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.146371 12349 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.147184  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:06.147998  8575 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:06.134606 12635 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:06.148595 12635 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:06.149526 12635 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.149971  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:06.151013  8584 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:06.117090 12521 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:03:06.152954 12521 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:06.153010  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:06.153453 12521 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:06.153680  8731 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:06.154299 12521 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.155637 12521 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:06.151854 12577 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:06.118222 12724 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:03:06.156018 12349 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.135525 12572 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:06.157708  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:06.158993 12715 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.168241 12739 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:06.183180 12715 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.134289 12709 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:06.183657 12739 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.185528 12667 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:06.186475 12667 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:06.160681 12735 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:06.186846 12667 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:06.163219  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:06.187733 12667 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:06.188918 12667 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:06.189452  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:06.189716 12577 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:06.190056  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:06.190373 12577 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:06.171605  8729 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:06.193555  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 754) took 2321 ms. Trace:
I20260709 13:03:06.194098  8733 rpcz_store.cc:276] 0709 13:03:03.872057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.872179 (+   122us) service_pool.cc:224] Handling call
0709 13:03:03.872507 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.872701 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:03.872724 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.958617 (+2085893us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b2c000b2cd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:05.959063 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.091715 (+132652us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:06.091773 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.091847 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.193245 (+101398us) raft_consensus.cc:1685] finished
0709 13:03:06.193268 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.193496 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24489728,"thread_start_us":132299,"threads_started":1,"wal-append.queue_time_us":231872}
I20260709 13:03:06.196424  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:06.173769  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:06.199898  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:06.200493  8730 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:03:06.200171  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:06.176107 12268 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:06.176362 12342 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:06.172353 12594 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 13:03:06.201865  8733 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:06.203066 12594 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:06.203491 12594 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:06.204380 12594 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:06.206283  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:06.172123 12532 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:03:06.207883 12268 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:06.208244 12594 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:06.208277 12268 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:06.209496 12268 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:06.210688 12268 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:06.211621  9324 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:06.211707  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:06.213018 12746 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.559s	user 0.000s	sys 0.002s
I20260709 13:03:06.113586 12629 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.101s
I20260709 13:03:06.214357  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:06.214987 12629 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:03:06.215389 12629 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:06.216298 12629 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.171092 12572 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:06.176982  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:06.220059  8728 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:06.177791 12724 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:06.178337  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:06.179397  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:06.221272 12724 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:06.222357  8726 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:03:06.193617  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:06.222584 12724 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.176717 12717 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:06.208568  8652 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:03:06.234365  8731 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:06.227165  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:06.239518  8728 raft_consensus.cc:2410] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:03:06.239619  8729 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:06.240339  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:06.226660  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:06.208535 12752 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.540s	user 0.002s	sys 0.000s
I20260709 13:03:06.246603  8727 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
W20260709 13:03:06.247102  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.234755  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:06.225106  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.242879  9327 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:03:06.244108  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:06.252524  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:06.262619  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:06.263748  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:06.268674  9327 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:06.268257  8730 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:06.270845  9329 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:06.282218  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:06.246506 12573 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.037s
I20260709 13:03:06.221055 12572 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.274087  8728 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:03:06.276517  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:06.277499  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:06.257292 12558 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:06.376037  9329 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:06.262087 12629 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:06.262153 12744 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:06.257032 12735 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:06.246196 12555 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:03:06.280308  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:06.406602  8724 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:06.412343  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.330734  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:06.276960 12721 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:06.270154 12724 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:06.270140 12175 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 13:03:06.327108 12715 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:06.277420 12546 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.275534 12717 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:03:06.275417  8468 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:03:06.480701 12717 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:06.480875  8468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 205685us
I20260709 13:03:06.481223 12717 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:06.480724 12721 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:06.482529 12649 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.483311 12555 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:06.483846  8726 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:06.483987 12555 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.485070 12649 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:06.360893  9324 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:06.270090 12765 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:06.378460  8581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.016s	sys 0.049s
I20260709 13:03:06.377723 12558 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:06.530324  8581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.016s	sys 0.049s
W20260709 13:03:06.471836  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.273466  8625 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:06.532155  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 258734us
I20260709 13:03:06.533573  8625 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:06.446596 12744 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:06.490885 12751 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.832s	user 0.001s	sys 0.001s
I20260709 13:03:06.530426 12715 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:06.559303 12718 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:03:06.561148  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.630820  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:03:06.575316  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.037s	sys 0.048s
W20260709 13:03:06.575747  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.037s	sys 0.048s
I20260709 13:03:06.576386  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:06.590189  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:06.563376 12268 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:03:06.602557  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet a6c36d18684a44799bfb89a6d20e4fd7: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:03:06.482573 12721 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.577821 12290 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.567308 12718 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:06.616223  8622 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.015s	sys 0.039s
W20260709 13:03:06.640430  8622 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.015s	sys 0.039s
W20260709 13:03:06.633927 12664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.005s	sys 0.020s
I20260709 13:03:06.635773 12558 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:06.641661 12664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.006s	sys 0.020s
I20260709 13:03:06.642004 12558 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:06.642419 12558 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:06.642786 12558 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:06.567366 12342 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.067s
W20260709 13:03:06.666047  8577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.013s	sys 0.041s
W20260709 13:03:06.666678  8577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.014s	sys 0.041s
I20260709 13:03:06.563808 12765 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:06.640826  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 525401us
I20260709 13:03:06.648592 12290 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:06.690368 12467 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.010s	sys 0.032s
W20260709 13:03:06.724359  9326 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.010s	sys 0.016s
W20260709 13:03:06.729605  9326 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.010s	sys 0.016s
I20260709 13:03:06.684813 12342 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:06.646123 12718 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:06.738183  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 503415us
I20260709 13:03:06.664808 12715 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:06.730542 12467 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.013s	sys 0.033s
W20260709 13:03:06.743395  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.733815 12342 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:06.742550 12665 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.084s	user 0.011s	sys 0.011s
I20260709 13:03:06.736976 12555 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:06.757076 12768 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:06.758889  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2636) took 1028 ms. Trace:
I20260709 13:03:06.759306  8651 rpcz_store.cc:276] 0709 13:03:05.730070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.030087 (+300017us) service_pool.cc:224] Handling call
0709 13:03:06.030413 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.030946 (+   533us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:06.050322 (+ 19376us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.278042 (+227720us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:06.278502 (+   460us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.278636 (+   134us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:06.278667 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.278730 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.758649 (+479919us) raft_consensus.cc:1685] finished
0709 13:03:06.758669 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.758824 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3115,"prepare.queue_time_us":19187,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":4163,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":54289,"thread_start_us":201064,"threads_started":1,"wal-append.queue_time_us":592447}
I20260709 13:03:06.678874 12744 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:06.772771  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.010s	sys 0.017s
W20260709 13:03:06.748569 12689 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.365s	user 0.011s	sys 0.017s
W20260709 13:03:06.766069 12594 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.005s
I20260709 13:03:06.739928  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:06.777609  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:06.777633  8576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 1971) took 2313 ms (client timeout 1652 ms). Trace:
W20260709 13:03:06.796852  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.010s	sys 0.017s
W20260709 13:03:06.796481  8576 rpcz_store.cc:269] 0709 13:03:04.464052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.464981 (+   929us) service_pool.cc:224] Handling call
0709 13:03:06.777284 (+2312303us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b54003a4654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:06.777551 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96273280}
W20260709 13:03:06.797614  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 539609us
I20260709 13:03:06.798728  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:06.778185  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:06.749514 12665 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.011s	sys 0.011s
W20260709 13:03:06.810892  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 71064.3us
W20260709 13:03:06.809849  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59526.6us
W20260709 13:03:06.798610 12689 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.415s	user 0.011s	sys 0.018s
W20260709 13:03:06.814013 12769 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 13:03:06.814160 12290 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.811506 12721 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:06.816298 12579 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.817371 12579 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:06.820022 12514 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:06.808523 12546 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:06.820945 12546 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.823001 12546 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:06.824170 12765 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.824837 12765 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:06.824995 12723 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.667s	user 0.000s	sys 0.010s
I20260709 13:03:06.825378 12765 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:06.827939  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2626) took 1853 ms (client timeout 1503 ms). Trace:
I20260709 13:03:06.828605  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 762) took 1930 ms. Trace:
I20260709 13:03:06.829010  8725 rpcz_store.cc:276] 0709 13:03:04.898455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.901884 (+  3429us) service_pool.cc:224] Handling call
0709 13:03:04.902351 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:04.902803 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:04.902843 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:04.903408 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.090553 (+187145us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:05.090612 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.090711 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.824381 (+1733670us) spinlock_profiling.cc:243] Waited 9.81 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:06.828238 (+  3857us) raft_consensus.cc:1685] finished
0709 13:03:06.828260 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.828536 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3206,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"spinlock_wait_cycles":22559488,"thread_start_us":186762,"threads_started":1,"wal-append.queue_time_us":254528}
W20260709 13:03:06.828435  8653 rpcz_store.cc:269] 0709 13:03:04.975098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.979367 (+  4269us) service_pool.cc:224] Handling call
0709 13:03:06.827761 (+1848394us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b5400410ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:06.827868 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":973236992}
I20260709 13:03:06.831398 12171 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:06.832279  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 963) took 1473 ms. Trace:
I20260709 13:03:06.832747  9331 rpcz_store.cc:276] 0709 13:03:05.358485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.358743 (+   258us) service_pool.cc:224] Handling call
0709 13:03:05.369106 (+ 10363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.505659 (+136553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.832019 (+1326360us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540001e354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:06.832215 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":237000960}
W20260709 13:03:06.833540  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.835328 12615 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:06.835462 12772 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:03:06.836541 12579 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:06.836750 12615 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:06.837070 12579 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:06.838107 12579 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.812831 12342 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.840186 12342 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:06.844007 12748 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.019s	sys 0.044s
W20260709 13:03:06.844477 12748 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.019s	sys 0.044s
I20260709 13:03:06.845958  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2639) took 1005 ms. Trace:
I20260709 13:03:06.846534  8658 rpcz_store.cc:276] 0709 13:03:05.840048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.844268 (+  4220us) service_pool.cc:224] Handling call
0709 13:03:06.845610 (+1001342us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b5400139f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:06.845852 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1748304640}
I20260709 13:03:06.859608 12633 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:06.860299 12349 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:06.862898  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 626) took 1174 ms. Trace:
I20260709 13:03:06.863449  8659 rpcz_store.cc:276] 0709 13:03:05.688258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.699009 (+ 10751us) service_pool.cc:224] Handling call
0709 13:03:05.721742 (+ 22733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.892440 (+170698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.862569 (+970129us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400352ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:06.862819 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369200512}
I20260709 13:03:06.860913 12349 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:06.877760 12615 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.010s
I20260709 13:03:06.810729 12175 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:06.890336  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 729) took 1313 ms (client timeout 1612 ms). Trace:
I20260709 13:03:06.890738 12175 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:06.891003  9323 rpcz_store.cc:269] 0709 13:03:05.577015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.577163 (+   148us) service_pool.cc:224] Handling call
0709 13:03:06.890070 (+1312907us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54004de254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:06.890236 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":352667520}
I20260709 13:03:06.893268  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2633) took 1194 ms. Trace:
I20260709 13:03:06.893908  8655 rpcz_store.cc:276] 0709 13:03:05.699397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.719606 (+ 20209us) service_pool.cc:224] Handling call
0709 13:03:05.719912 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.947398 (+227486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.947445 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:05.947502 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:05.947560 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.892945 (+945385us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:06.893010 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.893187 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522546048}
I20260709 13:03:06.895650 12349 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:06.901350  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 627) took 1173 ms. Trace:
I20260709 13:03:06.902012  8656 rpcz_store.cc:276] 0709 13:03:05.728033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.998072 (+270039us) service_pool.cc:224] Handling call
0709 13:03:05.998309 (+   237us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.077824 (+ 79515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.077868 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:06.078397 (+   529us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.281265 (+202868us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:06.281307 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.900819 (+619512us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b54000596d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:06.900886 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.900952 (+    66us) raft_consensus.cc:1685] finished
0709 13:03:06.900973 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.901265 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2033,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":100,"spinlock_wait_cycles":181941632,"thread_start_us":202420,"threads_started":1,"wal-append.queue_time_us":569893}
W20260709 13:03:06.958846  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 72143.3us
W20260709 13:03:06.961412  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.809010 12735 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:06.960112  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:06.972692  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.987429  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:06.988466  8655 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:06.820079 12717 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:06.934536 12755 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.003s	sys 0.026s
I20260709 13:03:06.998077  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:06.999156  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.003239  9331 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:06.969547 12748 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:07.004763  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.005939  8659 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:03:07.008024  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48016.6us
I20260709 13:03:07.009356  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:07.015241  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.016987  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.018921  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:06.997447 12755 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.004s	sys 0.026s
W20260709 13:03:06.997804 12521 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.825s	user 0.010s	sys 0.002s
I20260709 13:03:06.961257 12765 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.036921  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.019013 12528 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:07.035074  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:07.069780  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:07.120494  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.018607 12610 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.018817 12555 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:07.018616 12748 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:07.115021  8580 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:07.127065  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.130787  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.022290 12635 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.115s
I20260709 13:03:07.118917 12497 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:07.057152 12175 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:07.120661 12667 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.931s	user 0.016s	sys 0.000s
I20260709 13:03:07.118769 12547 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 13:03:07.056234 12721 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 13:03:07.122328 12779 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.071673 12594 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 13:03:07.045677 12577 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.110s
W20260709 13:03:07.162635  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 350839us
I20260709 13:03:07.166754  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.159659  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.181303  8732 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260709 13:03:07.181445  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:07.181922  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 605522us
I20260709 13:03:07.182176  8732 pending_rounds.cc:85] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:03:07.182605  8732 pending_rounds.cc:107] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 13:03:07.182925  8732 raft_consensus.cc:2889] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:07.182770  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:07.181422  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.142618 12497 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:07.184964  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:07.153790 12558 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.001s
I20260709 13:03:07.195906  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:07.149119 12780 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.147547 12735 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.176689 12779 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.196285 12761 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:07.217331  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:07.218601  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:07.221051 12528 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:07.176342 12594 consensus_queue.cc:1243] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:07.197781  9326 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:07.194423  8726 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:07.137683 12610 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.194422  8729 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 13:03:07.233027  8729 pending_rounds.cc:85] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 4
I20260709 13:03:07.233377  8729 pending_rounds.cc:107] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 13:03:07.233839  8729 raft_consensus.cc:2889] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:07.211792  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:07.235131  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50250.4us
I20260709 13:03:07.149341 12290 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.160449 12667 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.209741  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.231556  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:07.235831  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:07.240885  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:07.196022 12497 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.208011 12555 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:07.207239 12717 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.214991 12779 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.244853 12667 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:07.230782 12714 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.853s	user 0.015s	sys 0.027s
I20260709 13:03:07.203078 12789 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.256747 12656 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:07.233852 12665 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:07.278216 12572 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.280586 12531 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:07.375777 12789 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.382409 12665 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.289043  9324 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:07.238972 12268 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.022s
W20260709 13:03:07.299371 12785 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:03:07.202894 12790 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 13:03:07.224649 12558 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.247257 12755 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:03:07.308729  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 345283us
W20260709 13:03:07.311214  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:07.377432 12784 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260709 13:03:07.234053 12655 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.228668 12783 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:03:07.407051 12743 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.407841 12520 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.407600 12743 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.408286 12667 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.211859 12721 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 13:03:07.408362 12520 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.208477 12780 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:07.413542  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.414098 12780 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.414225 12520 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.212628 12786 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260709 13:03:07.239886 12718 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.239200 12290 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.333390  8655 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:07.257534 12782 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 13:03:07.206074 12748 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:07.330983 12775 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 13:03:07.300729 12693 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
W20260709 13:03:07.346029  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:07.428653  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:07.429256  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.234689 12709 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:03:07.287868 12778 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 13:03:07.276595  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:07.251420  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2585) took 1572 ms. Trace:
I20260709 13:03:07.447976  8577 rpcz_store.cc:276] 0709 13:03:05.679302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.679471 (+   169us) service_pool.cc:224] Handling call
0709 13:03:05.698551 (+ 19080us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.097969 (+399418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.097998 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:06.098524 (+   526us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.667882 (+569358us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:07.190625 (+522743us) log.cc:844] Serialized 14 byte log entry
0709 13:03:07.250600 (+ 59975us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.250803 (+   203us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.250891 (+    88us) raft_consensus.cc:1685] finished
0709 13:03:07.250940 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.251279 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69727,"prepare.queue_time_us":671625,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":156276,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":31637,"thread_start_us":1444774,"threads_started":3,"wal-append.queue_time_us":773984}
I20260709 13:03:07.372110  8578 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 13:03:07.370167 12714 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.992s	user 0.015s	sys 0.027s
W20260709 13:03:07.451445  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 174949us
I20260709 13:03:07.452267  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:07.202080 12720 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.817s	user 0.009s	sys 0.036s
I20260709 13:03:07.386773  9324 raft_consensus.cc:606] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:07.370417 12342 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.253628 12616 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:07.264793 12528 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:07.422333  9322 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:07.421357  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.459095 12342 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.459955  9326 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:07.423599 12633 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.439960  8650 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:07.425256 12665 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.461645  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:07.464434  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:07.472057  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 168882us
W20260709 13:03:07.457653 12720 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.080s	user 0.009s	sys 0.037s
W20260709 13:03:07.473373  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 261610us
I20260709 13:03:07.446323 12782 consensus_queue.cc:1243] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:07.476938  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.477705  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:07.479161  8655 raft_consensus.cc:2376] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 10. Re-sending same reply.
I20260709 13:03:07.481648  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 635) took 1272 ms. Trace:
I20260709 13:03:07.482081  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:07.478662  8650 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:07.485890  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:07.486838  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.488148  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.482051  8654 rpcz_store.cc:276] 0709 13:03:06.208850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.212761 (+  3911us) service_pool.cc:224] Handling call
0709 13:03:06.213079 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.213415 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.213444 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:06.213899 (+   455us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.696915 (+483016us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:06.696989 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.697058 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.481454 (+784396us) raft_consensus.cc:1685] finished
0709 13:03:07.481475 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.481590 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1097,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":83,"thread_start_us":482698,"threads_started":1,"wal-append.queue_time_us":779948}
I20260709 13:03:07.463521 12748 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:07.462357 12555 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:03:07.473227 12773 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.271s	user 0.000s	sys 0.001s
I20260709 13:03:07.468418 12786 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:07.494421  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.499166  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:07.495774  8650 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
I20260709 13:03:07.423781 12655 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.497810  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 637) took 1248 ms. Trace:
I20260709 13:03:07.515053  8652 rpcz_store.cc:276] 0709 13:03:06.249299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.250359 (+  1060us) service_pool.cc:224] Handling call
0709 13:03:06.250698 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.251068 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.251097 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:06.251541 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.838903 (+587362us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b18004794a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:07.229309 (+390406us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:07.232354 (+  3045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.232470 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.485355 (+252885us) raft_consensus.cc:1685] finished
0709 13:03:07.485383 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.497743 (+ 12360us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":7043,"spinlock_wait_cycles":9874304,"thread_start_us":390117,"threads_started":1,"wal-append.queue_time_us":1121120}
I20260709 13:03:07.483734 12796 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:07.493966 12555 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.518438 12555 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:07.523057  9324 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:07.523736  9324 raft_consensus.cc:606] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:07.523923  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.500211  8659 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:07.510509  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.524616  8654 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:07.497159  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.525281  8650 raft_consensus.cc:2410] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:03:07.518391 12796 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.483561 12779 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:07.517047  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:07.529495  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.532579  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.533869  8657 raft_consensus.cc:2410] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:03:07.493389 12720 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:07.492004 12610 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.498320 12786 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.473680 12633 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.545497 12342 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.545958 12633 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.547557 12743 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.478408 12528 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
W20260709 13:03:07.541764 12795 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 13:03:07.568050  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2173) took 1319 ms. Trace:
I20260709 13:03:07.568495  8731 rpcz_store.cc:276] 0709 13:03:06.248397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.248805 (+   408us) service_pool.cc:224] Handling call
0709 13:03:06.266613 (+ 17808us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.273306 (+  6693us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:06.727608 (+454302us) log.cc:844] Serialized 14 byte log entry
0709 13:03:07.236325 (+508717us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:07.252189 (+ 15864us) log.cc:844] Serialized 26 byte log entry
0709 13:03:07.252398 (+   209us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:07.252445 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.252533 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.567834 (+315301us) raft_consensus.cc:1685] finished
0709 13:03:07.567855 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.567987 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":109342,"prepare.queue_time_us":103199,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":933955,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":26385,"spinlock_wait_cycles":7892736,"thread_start_us":862968,"threads_started":2,"wal-append.queue_time_us":592563}
I20260709 13:03:07.570248  9327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea2142382a5d4439915e07c70837c688"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:07.541879 12290 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.543076 12558 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.513576 12532 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:07.553962 12786 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.557554 12718 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.529676 12779 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.532245 12717 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:07.577773  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.559286 12721 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.579095 12497 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.580632  8577 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:07.581502 12497 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:07.570873  9327 raft_consensus.cc:606] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:07.582191  9327 raft_consensus.cc:614] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:07.528386 12796 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.583333  8651 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:07.583680  8651 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:07.584474  8651 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.577569 12717 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:07.575503  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 168875us
I20260709 13:03:07.586206  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:07.570900 12743 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.579123 12721 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.574234 12532 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:07.592609  8575 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:03:07.593473  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.593820  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.594095  8579 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:07.577939 12633 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.587975  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:03:07.588215  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:07.554571  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:07.588481  8583 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:07.585974 12717 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:07.600440 12765 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:07.600997 12765 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:07.601752  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.601972 12765 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.594317 12532 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.594885  8575 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:07.565263  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.577668 12779 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.599507  8658 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:07.604758 12786 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:07.607604  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 78279.3us
I20260709 13:03:07.587812 12809 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.613199  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:07.616655  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:07.618831  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.609978 12718 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:07.610324 12633 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.627000 12718 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.627612 12718 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.628396 12633 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.631902 12721 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.625231 12342 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 13:03:07.631846 12555 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.635217 12555 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.636301  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:07.592834 12743 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.637013  9324 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:07.638350 12743 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.638579  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 964) took 2024 ms. Trace:
I20260709 13:03:07.623121 12790 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.016s
I20260709 13:03:07.639103  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:07.638988  9322 rpcz_store.cc:276] 0709 13:03:05.614552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.614681 (+   129us) service_pool.cc:224] Handling call
0709 13:03:06.892353 (+1277672us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:07.459534 (+567181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.637597 (+178063us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b54004de254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:07.638392 (+   795us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54004de250. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:07.638523 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4153585792}
I20260709 13:03:07.640947 12342 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:07.605718 12755 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.621589 12809 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.641762 12755 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.624905 12175 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.643594 12809 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.643857 12175 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.645385 12786 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.645802 12744 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:07.646095 12175 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.646157 12744 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:07.647261 12497 raft_consensus.cc:993] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:03:07.647260 12744 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.647866 12497 raft_consensus.cc:1081] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:03:07.648375 12744 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:07.650377 12718 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.652493 12635 raft_consensus.cc:993] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:03:07.660089  8575 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:07.661470  8658 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.662040  8582 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:07.663365  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:07.664810  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.621229 12290 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:07.615549 12815 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:07.654623 12780 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:07.658811  9327 raft_consensus.cc:1217] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Deduplicated request from leader. Original: 10.8->[11.9-11.9]   Dedup: 11.9->[]
I20260709 13:03:07.667454  9327 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:03:07.667598  8582 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:07.672158  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:07.674156  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.676599  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.676968  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.682921 12532 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.685719  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:07.625061 12720 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:07.656059 12342 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.691604  8581 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:07.695029  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.695784  8577 raft_consensus.cc:2410] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:03:07.667297 12780 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:07.701604 12577 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:07.702015 12577 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:07.687314 12635 raft_consensus.cc:1081] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:03:07.687352 12788 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:07.703316  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2591) took 2017 ms. Trace:
I20260709 13:03:07.703413 12788 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.703759  8578 rpcz_store.cc:276] 0709 13:03:05.686584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.686966 (+   382us) service_pool.cc:224] Handling call
0709 13:03:05.705499 (+ 18533us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:07.450683 (+1745184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.703072 (+252389us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54001694d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:07.703261 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3831401728}
I20260709 13:03:07.706236 12788 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.703953 12532 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.701164 12780 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:07.707966 12797 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.334s	user 0.001s	sys 0.001s
I20260709 13:03:07.708401 12780 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:07.712905  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.713095  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:07.697777  8581 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:07.717259 12789 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.718974  9322 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: [3.5-7.7]
I20260709 13:03:07.725895  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.726604  8581 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:07.693938 12342 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.730904  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.731446  8581 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:07.733493  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.741123  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.741868  8577 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:07.741693 12796 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.743515  8580 raft_consensus.cc:2368] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8.
W20260709 13:03:07.745867  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 510012us
I20260709 13:03:07.747360 12163 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:07.747740 12163 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.748063 12163 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:07.748129  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:07.749804  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 518256us
I20260709 13:03:07.750028 12655 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.751816 12655 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:07.752214  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.752610  8657 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:07.752867  8732 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
W20260709 13:03:07.755538 12793 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.371s	user 0.004s	sys 0.000s
I20260709 13:03:07.755736 12649 raft_consensus.cc:2764] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:03:07.756269 12486 raft_consensus.cc:2764] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:03:07.757068 12486 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:07.758373  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.758821  8731 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:07.759228  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.766906  8730 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:07.762857  8728 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:07.763160 12755 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.768041  8728 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:07.770481  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.772087  8577 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:07.768949 12717 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.765334 12486 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:07.774446 12486 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.775571 12486 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.778278 12779 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.778884  8651 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.779480  8583 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:07.781692  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 304274us
I20260709 13:03:07.770296  8728 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.782644  8728 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:07.782876  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:07.784330 12616 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:07.785346 12656 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:07.785980  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.786274 12656 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.787071  8729 raft_consensus.cc:2410] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:03:07.788479 12163 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.785184  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:07.790867  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.792614 12616 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:07.794366  8577 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:34698
I20260709 13:03:07.795114  8577 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:07.794770 12788 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:07.795506  8577 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.796741 12788 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:07.797729  8583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:34698
I20260709 13:03:07.807520  8583 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:07.807888  8583 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.799680  8650 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:07.798759 12163 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:07.794438  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:07.800755  8656 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:07.801739  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.803761  8655 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:07.812880  8728 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 13:03:07.813382  8728 pending_rounds.cc:85] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 9
I20260709 13:03:07.813722  8728 pending_rounds.cc:107] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 10.10
W20260709 13:03:07.813844  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.814296  8728 raft_consensus.cc:2889] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:07.814779  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.791054  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:07.822171  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.804953  8577 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:07.831027  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.838068  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.832134  8727 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:07.863456  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.805598 12796 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.860627  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:07.873914  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:07.874781  8655 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:07.878471  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.880563  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.824786 12779 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.886603  8575 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:07.890468  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.891252  8584 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:07.892015  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.892695  8584 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:07.893258  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.894511  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.826454 12744 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:07.820194 12532 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.902709  8577 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.903260  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.915700  8582 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.883380  8583 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:07.866499 12822 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.945170  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.888827 12664 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:07.948534  8582 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:07.887008 12655 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:07.819782 12514 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:07.914642 12720 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.905122  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.955204  8584 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:07.956089  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.956861  8584 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:07.966938  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.969921  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.925701  8655 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:07.927537 12821 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:03:07.972903  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:07.930114  8583 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:07.974893  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.975520  8583 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:07.931818  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:07.976086  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.976944  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:07.932783  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:07.932796  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.978032  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.978212  8732 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:07.978727  8584 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:07.934222 12354 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:07.985946  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:07.986817  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.987488  8732 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:07.939965 12616 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:07.940356 12532 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.992496 12532 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:07.993124  8730 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.941845  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:07.942615  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:08.001173  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.829460 12547 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20260709 13:03:07.952454  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:08.001900  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:08.002167  8577 raft_consensus.cc:2376] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9. Re-sending same reply.
I20260709 13:03:07.735262 12577 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.952941  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:07.954711 12664 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:07.947588 12796 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.972216  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:07.948845 12822 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:08.017728  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 187093us
I20260709 13:03:07.987850  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:07.956341 12514 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:07.997087 12765 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:07.971110  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:08.013027  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60170.5us
I20260709 13:03:08.046828  8730 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:08.040742  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:08.013698  8582 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:07.984092  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:08.021624  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:08.022763  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:08.050544  8582 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:03:07.908641  8651 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 13:03:08.051978  8651 pending_rounds.cc:85] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 4
I20260709 13:03:07.970315 12720 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.053485 12788 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:08.053821 12514 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:08.053916 12788 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:08.054457 12788 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:08.054860 12720 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:08.002983 12780 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:08.056856 12822 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:08.052215  8651 pending_rounds.cc:107] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 13:03:08.026345 12796 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.051597  8584 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:07.985839 12717 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:08.058187 12819 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260709 13:03:08.058823 12735 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:08.057977  8651 raft_consensus.cc:2889] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:08.059259 12735 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:08.059557 12796 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:08.059773  8651 pending_rounds.cc:107] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 13:03:08.060178 12735 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:08.060768  8651 raft_consensus.cc:2889] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:08.061435  8651 pending_rounds.cc:107] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 13:03:08.061879  8651 raft_consensus.cc:2889] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:08.066184  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:08.069193 12744 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.069968 12744 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:08.072080 12791 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 13:03:08.073315 12735 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:08.074996  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:08.074806  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:08.075788  8580 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:08.076031  8651 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:08.076946  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:08.077096  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:08.078177  8650 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:03:08.078085  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:08.078753  8655 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:08.079304  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:08.079854  8650 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:08.080338  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:08.077716  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:07.958828 12655 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:08.084350  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 190580us
W20260709 13:03:08.085208  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63723us
I20260709 13:03:08.086822  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:08.088474  8651 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:08.085084  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:08.095475  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:08.097012  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:08.099052  8584 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:08.099385  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:08.099993  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.106945  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:08.108636  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:08.108817  8584 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:07.987079 12342 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:08.110805 12486 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:08.113472  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:08.115957  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:08.108666 12739 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:08.072098 12744 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:08.108922 12532 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 13:03:08.074831 12664 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.140147 12335 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:08.094830 12616 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 13:03:08.140622 12175 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:08.116837 12486 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:08.155108 12486 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:08.111001 12171 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:08.138607  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:08.108872 12822 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:08.103190 12720 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:08.103740 12717 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:03:08.116873 12815 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:08.156065 12829 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:08.111672 12635 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 13:03:08.150496 12739 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:08.110219 12514 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:03:08.164919 12815 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:08.180050 12815 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:08.161849 12822 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:08.174409 12739 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:08.197728 12835 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:08.200220 12835 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:08.201570 12835 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.205272 12655 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:08.190874 12829 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 13:03:08.195808 12788 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:08.212495 12788 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:08.191793 12822 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:08.232062 12815 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:08.234990 12815 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:08.242750 12739 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:08.253080 12822 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.255885 12822 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:08.287506  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.289776  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.292529  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 308328us
I20260709 13:03:08.294588  8576 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:08.297191  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:08.302230  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:08.303557  8582 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:08.304085  8584 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:08.101930  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:08.318688  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:08.320312  8651 raft_consensus.cc:2410] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:03:08.320395 12780 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.321530 12780 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:08.324842  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 186108us
I20260709 13:03:08.335685 12573 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.335743 12780 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:08.342274 12724 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:08.344609 12342 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:08.346443 12171 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:08.344630 12765 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.433552 12573 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.436383 12573 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:08.437477 12739 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:08.438802 12796 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.439848 12796 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.445660  8728 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 6. Current term is 8. Ops: []
I20260709 13:03:08.449227  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:08.449640 12786 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:08.450296  8729 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:08.453166  8730 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:08.450296 12786 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:08.455315 12836 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.050s	user 0.000s	sys 0.002s
I20260709 13:03:08.455695 12786 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:08.456243  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 335939us
I20260709 13:03:08.432353 12171 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:08.480257 12765 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:08.482064  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.483561  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 161261us
I20260709 13:03:08.488243  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:08.490105 12788 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:08.502797 12635 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:08.507496 12739 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:08.513064 12779 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.519251 12779 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:08.521916 12779 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:08.549286  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.543857 12796 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:08.557634  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.551916 12765 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:08.637360 12848 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260709 13:03:09.001706  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.034550 12786 consensus_meta.cc:287] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.578s	user 0.011s	sys 0.003s
W20260709 13:03:09.123101  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.179304 12843 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.640s	user 0.005s	sys 0.006s
W20260709 13:03:09.224248 12831 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.038s	user 0.000s	sys 0.020s
W20260709 13:03:09.231849  8583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.163s	user 0.006s	sys 0.007s
W20260709 13:03:09.232394  8583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.164s	user 0.006s	sys 0.007s
W20260709 13:03:09.258374  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 405029us
I20260709 13:03:09.262346  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:09.262665  8650 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.172s	user 0.009s	sys 0.014s
I20260709 13:03:09.263233  8650 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:09.264704  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2475) took 1193 ms. Trace:
I20260709 13:03:09.265373  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:09.265465  8650 rpcz_store.cc:276] 0709 13:03:08.071015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.079092 (+  8077us) service_pool.cc:224] Handling call
0709 13:03:09.264527 (+1185435us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:09.271085  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:09.273667  8696 leader_election.cc:400] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8.
W20260709 13:03:09.305083  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 817046us
I20260709 13:03:09.320604  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2641) took 1522 ms. Trace:
I20260709 13:03:09.325173  8658 rpcz_store.cc:276] 0709 13:03:07.798367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.799505 (+  1138us) service_pool.cc:224] Handling call
0709 13:03:09.317369 (+1517864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:09.317910 (+   541us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:09.317950 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:09.318022 (+    72us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:09.318062 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:09.318157 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:09.320194 (+  2037us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540018ecd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:09.320371 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3489649920}
I20260709 13:03:09.324271 12483 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:09.326857 12483 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:09.328750 12483 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:09.330387 12663 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:09.331902 12483 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:09.332322 12663 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:09.334805 12483 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:09.335577 12483 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:09.336318 12483 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:09.352308 12663 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.358459 12850 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.359443 12850 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.362330 12483 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.363229 12850 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:09.365059 12483 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.372979  8579 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:09.354246 12663 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.380155 12663 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.392680  8579 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
W20260709 13:03:09.797313  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.803062 12850 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.803925 12850 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.806479 12850 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.819307 12483 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:09.819566 12853 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.819993 12483 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.820113 12853 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.822388 12483 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.823128 12853 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:09.828094  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.838564 12342 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:09.839248 12342 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:09.839725 12342 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:09.844395 12709 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.845228 12709 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:09.846261  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 1.744e+06us
I20260709 13:03:09.848029 12709 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.847926  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:09.849754  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 547703us
I20260709 13:03:09.851100  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:09.851866  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.852859  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:09.853042 12335 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:09.853935  8650 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 13:03:09.853708 12335 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.854317  8650 pending_rounds.cc:85] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 8
I20260709 13:03:09.854642  8650 pending_rounds.cc:107] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 13:03:09.854925  8650 raft_consensus.cc:2889] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:09.856153 12335 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:09.856297  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:09.856745 12335 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.857318 12335 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.860684  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:09.862156  8657 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:09.862042 12532 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:09.863214 12268 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:09.863659 12268 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:09.864886  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:09.864111 12268 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:09.866788  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:09.867342  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:09.863808 12532 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 13:03:09.867244 12171 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:09.868309  8657 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:09.868503 12171 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:09.869051 12171 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:09.869204 12335 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:09.870862  9296 connection.cc:256] RPC call timeout handler was delayed by -1.0980447430000595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
I20260709 13:03:09.871520 12629 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:09.872224 12629 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.872817 12497 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.873488 12497 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.874696 12629 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:09.875746 12497 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:09.877224  9331 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:09.878367  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 800) took 2051 ms. Trace:
I20260709 13:03:09.878486 12709 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.879603 12717 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:09.879232 12709 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.880177 12717 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:09.881511 12717 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.884981 12709 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.878850  8578 rpcz_store.cc:276] 0709 13:03:07.826973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.827129 (+   156us) service_pool.cc:224] Handling call
0709 13:03:07.887956 (+ 60827us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:07.888303 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:07.888322 (+    19us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:07.911020 (+ 22698us) log.cc:844] Serialized 26 byte log entry
0709 13:03:08.299000 (+387980us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:08.299126 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.299354 (+   228us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:09.877944 (+1578590us) raft_consensus.cc:1685] finished
0709 13:03:09.877962 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:09.878172 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36650,"raft.queue_time_us":21353,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":7121,"thread_start_us":387150,"threads_started":1,"wal-append.queue_time_us":530643,"wal-append.run_cpu_time_us":3068,"wal-append.run_wall_time_us":1003083}
I20260709 13:03:09.890182  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:09.897852 12268 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.901757 12268 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:09.903367 12342 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.906651 12342 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:09.911757  8657 raft_consensus.cc:2410] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:03:09.916507 12342 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.916981  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:09.920101  9327 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:09.920756  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:09.921170  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.921852  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:09.918044 12268 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:09.922443  9329 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:09.922637 12268 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.924566  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:09.925030  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:09.925180  9324 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:09.925701  9324 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:09.926198  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:09.926532  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:09.926646  9324 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.927819  9324 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:09.927035  9329 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:09.928668  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:09.928891  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 76223.3us
I20260709 13:03:09.928722  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:09.929605  8650 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:09.929797  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:09.930101  8651 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:09.925725  9322 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:09.931028 12467 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:09.931874  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:09.932411 12268 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:09.932987  8577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.873s	user 0.014s	sys 0.031s
I20260709 13:03:09.931154  9322 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:09.934717  8481 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 203) took 30104 ms (client timeout 15000 ms). Trace:
I20260709 13:03:09.935002 12532 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 13:03:09.935168  8481 rpcz_store.cc:269] 0709 13:02:39.809889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.811615 (+  1726us) service_pool.cc:224] Handling call
0709 13:03:09.908573 (+30096958us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:58.404014 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:58.470425 (+ 66411us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:58.479633 (+  9208us) write_op.cc:435] Acquired schema lock
0709 13:02:58.491107 (+ 11474us) tablet.cc:662] Decoding operations
0709 13:02:58.522797 (+ 31690us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:58.522932 (+   135us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:58.522973 (+    41us) tablet.cc:685] Acquiring locks for 11 operations
0709 13:02:58.524083 (+  1110us) tablet.cc:701] Row locks acquired
0709 13:02:58.524137 (+    54us) write_op.cc:260] PREPARE: finished
0709 13:02:58.524341 (+   204us) write_op.cc:270] Start()
0709 13:02:58.524487 (+   146us) write_op.cc:276] Timestamp: P: 1783602178524263 usec, L: 0
0709 13:02:58.524526 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 13:02:58.525536 (+  1010us) log.cc:844] Serialized 4358 byte log entry
0709 13:02:58.916136 (+390600us) op_driver.cc:464] REPLICATION: finished
0709 13:02:59.040792 (+124656us) write_op.cc:301] APPLY: starting
0709 13:02:59.040926 (+   134us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:59.063113 (+ 22187us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:59.077146 (+ 14033us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:59.166718 (+ 89572us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:59.167200 (+   482us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=22
0709 13:02:59.167242 (+    42us) write_op.cc:312] APPLY: finished
0709 13:02:59.167653 (+   411us) log.cc:844] Serialized 83 byte log entry
0709 13:02:59.167836 (+   183us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:59.167991 (+   155us) write_op.cc:454] Released schema lock
0709 13:02:59.168051 (+    60us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:59.168517 (+   466us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:59.168548 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":758113,"reported_tablets":27,"spinlock_wait_cycles":2304,"child_traces":[["op",{"apply.queue_time_us":124293,"apply.run_cpu_time_us":6138,"apply.run_wall_time_us":127711,"num_ops":11,"prepare.queue_time_us":342144,"prepare.run_cpu_time_us":23394,"prepare.run_wall_time_us":249592,"raft.queue_time_us":119494,"raft.run_cpu_time_us":10319,"raft.run_wall_time_us":74075,"replication_time_us":391397,"thread_start_us":425152,"threads_started":4,"wal-append.queue_time_us":227476,"wal-append.run_cpu_time_us":23779,"wal-append.run_wall_time_us":1657976}]]}
I20260709 13:03:09.933326  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:09.933059 12268 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:03:09.942790  8577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.882s	user 0.014s	sys 0.032s
I20260709 13:03:09.944295  9324 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:09.946420  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:09.942368 12268 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:09.949608 12268 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:09.933660 12709 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.944608 12720 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:09.952149 12467 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.952884 12467 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.946532  9322 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.955978  9322 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:08.554019 12486 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.919646 12342 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.960582 12720 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:09.960952  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:09.962095 12720 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.962894 12486 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:09.964136 12717 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:09.964618 12717 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:09.965015 12717 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:09.965828 12791 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:09.976269 12791 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:09.976702 12791 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:09.968034  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2492) took 2074 ms. Trace:
I20260709 13:03:09.951508 12709 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.977674  8728 rpcz_store.cc:276] 0709 13:03:07.893263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.909416 (+ 16153us) service_pool.cc:224] Handling call
0709 13:03:08.410490 (+501074us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:08.446408 (+ 35918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:09.967787 (+1521379us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54000e9f50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:09.967943 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":657855232}
W20260709 13:03:09.978979 12832 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.793s	user 0.004s	sys 0.011s
I20260709 13:03:09.979806 12709 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.975685  9327 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:09.980823 12171 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:09.950706 12532 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.982291 12532 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.983062 12709 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.983310 12171 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:09.983857 12709 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.984479 12532 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.985391  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:09.985714 12467 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.973871  9322 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:09.989811  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:09.994534  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.994661  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2012) took 2101 ms. Trace:
W20260709 13:03:09.995458  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.995288  8583 rpcz_store.cc:276] 0709 13:03:07.903924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.002558 (+ 98634us) service_pool.cc:224] Handling call
0709 13:03:08.018181 (+ 15623us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:08.067590 (+ 49409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:08.067622 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:08.068359 (+   737us) log.cc:844] Serialized 78 byte log entry
0709 13:03:09.232975 (+1164616us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:09.234545 (+  1570us) log.cc:844] Serialized 14 byte log entry
0709 13:03:09.994184 (+759639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:09.994329 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:09.994384 (+    55us) raft_consensus.cc:1685] finished
0709 13:03:09.994401 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:09.994601 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":765544,"prepare.queue_time_us":124964,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":74,"raft.queue_time_us":49450,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":116,"thread_start_us":1165009,"threads_started":2,"wal-append.queue_time_us":334613}
I20260709 13:03:09.998003 12783 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:09.997287 12342 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:09.998697 12783 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.002769 12755 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.003614 12755 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.007014  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.007869  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2013) took 2094 ms. Trace:
I20260709 13:03:10.009068 12817 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.010776 12817 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.008411  8577 rpcz_store.cc:276] 0709 13:03:07.913284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.002749 (+ 89465us) service_pool.cc:224] Handling call
0709 13:03:08.003060 (+   311us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:08.003397 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:08.003423 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:08.007926 (+  4503us) log.cc:844] Serialized 52 byte log entry
0709 13:03:09.943792 (+1935866us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:09.951477 (+  7685us) log.cc:844] Serialized 14 byte log entry
0709 13:03:10.007282 (+ 55805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.007440 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:10.007509 (+    69us) raft_consensus.cc:1685] finished
0709 13:03:10.007531 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.007791 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49409,"prepare.queue_time_us":7716,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":560,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":8827,"thread_start_us":1896318,"threads_started":3,"wal-append.queue_time_us":391341,"wal-append.run_cpu_time_us":3690,"wal-append.run_wall_time_us":1011237}
I20260709 13:03:10.007016 12615 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.013851 12817 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.014649 12817 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.015403 12817 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:09.977283 12577 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:10.018893  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.014371 12615 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.029373 12863 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.032775  9327 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.040405 12779 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.040796 12615 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.041353 12615 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.041154 12779 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.041915 12615 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:10.044308  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.044175  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.033594 12783 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.049038 12755 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.037657 12817 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.060582 12859 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.043624 12786 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:10.050339  8579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2006) took 2159 ms (client timeout 1758 ms). Trace:
W20260709 13:03:10.048749  8655 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.969s	user 0.021s	sys 0.008s
W20260709 13:03:10.054162  8580 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.754s	user 0.016s	sys 0.003s
W20260709 13:03:10.066720  8579 rpcz_store.cc:269] 0709 13:03:07.891338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.894199 (+  2861us) service_pool.cc:224] Handling call
0709 13:03:10.050091 (+2155892us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54003449d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:10.050245 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3397549952}
I20260709 13:03:10.056664  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:10.058825  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:10.082301  8549 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.502s (SENT)
I20260709 13:03:10.086702 12615 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.096482  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:10.096812  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:10.103372  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.095353 12786 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.099493  8575 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:10.110345 12455 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:10.067523 12783 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:10.104735  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.111855 12783 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.112389  9323 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:10.097033  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2494) took 2178 ms. Trace:
I20260709 13:03:10.066287 12755 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.087121 12709 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.100612 12817 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.113266  8727 rpcz_store.cc:276] 0709 13:03:07.918265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.919438 (+  1173us) service_pool.cc:224] Handling call
0709 13:03:07.919724 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:08.115711 (+195987us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:08.115757 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:08.115800 (+    43us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:08.115835 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.096559 (+1980724us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400065254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:10.096627 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.096843 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":450255232}
I20260709 13:03:10.114740  8655 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:10.101581 12859 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.113588  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2018) took 2082 ms. Trace:
I20260709 13:03:10.115522 12290 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.115779  8575 rpcz_store.cc:276] 0709 13:03:08.031276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.040397 (+  9121us) service_pool.cc:224] Handling call
0709 13:03:10.113333 (+2072936us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54003e7954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:10.113501 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4716873600}
I20260709 13:03:10.116106 12290 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.061682 12863 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.095551 12779 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.081255  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.060585 12467 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 13:03:10.085536  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.068320  8728 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:10.067730  8580 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.768s	user 0.017s	sys 0.003s
I20260709 13:03:10.134040  9324 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:10.134528  8658 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.060863 12862 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.120327 12171 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:10.140064  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:10.140149  8655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2476) took 2065 ms (client timeout 1631 ms). Trace:
I20260709 13:03:10.141167  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.119001 12863 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:10.147030  8622 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.758s (SENT)
I20260709 13:03:10.141938  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:10.145494  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 106831us
W20260709 13:03:10.138170  8513 debug-util.cc:398] Leaking SignalData structure 0x7b080009dc20 after lost signal to thread 7484
I20260709 13:03:10.147790  9331 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:10.124192 12783 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:10.148463  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08002f9380 after lost signal to thread 8478
I20260709 13:03:10.120553 12290 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:10.143855  8655 rpcz_store.cc:269] 0709 13:03:08.071265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.077891 (+  6626us) service_pool.cc:224] Handling call
0709 13:03:10.131709 (+2053818us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:10.151463  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:10.150619  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:10.117324 12859 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.153486  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 801) took 2193 ms. Trace:
W20260709 13:03:10.154609  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08001bb360 after lost signal to thread 8516
I20260709 13:03:10.119689 12635 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.028s
W20260709 13:03:10.157214  8513 debug-util.cc:398] Leaking SignalData structure 0x7b080032e1e0 after lost signal to thread 8530
I20260709 13:03:10.157795  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.113771 12755 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.177364  9323 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:10.131809 12268 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.179538 12755 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.179605 12497 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:10.083546 12791 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.118906 12817 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.153075 12615 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.168176  8725 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:03:10.170773  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 88531.4us
I20260709 13:03:10.155179 12577 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:10.173807  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.175138  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.149017 12786 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:10.176380 12717 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.155844  8580 rpcz_store.cc:276] 0709 13:03:07.960476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.076593 (+116117us) service_pool.cc:224] Handling call
0709 13:03:08.076945 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:08.095826 (+ 18881us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:08.095857 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:08.298864 (+203007us) log.cc:844] Serialized 26 byte log entry
0709 13:03:10.131530 (+1832666us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:10.131597 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.140061 (+  8464us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:10.144375 (+  4314us) raft_consensus.cc:1685] finished
0709 13:03:10.144389 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.153430 (+  9041us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18577,"raft.queue_time_us":26293,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":2524,"thread_start_us":1829596,"threads_started":1,"wal-append.queue_time_us":168503,"wal-append.run_cpu_time_us":2772,"wal-append.run_wall_time_us":1013806}
I20260709 13:03:10.143401 12171 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:10.189319  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.190815  8657 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.196731  8651 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:10.197065 12615 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:10.197567 12714 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 13:03:10.197774 12791 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.235550  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.198623 12796 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.199939 12817 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.200301 12268 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.200526  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.201243  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.201606 12779 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:10.193717 12862 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.202453 12783 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:09.336333  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:03:10.220109  8549 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.514s (SENT)
I20260709 13:03:10.222952  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.224478 12455 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:10.225760  8478 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 6, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 6 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:10.198263 12863 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:10.238710  8727 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:10.214732 12577 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:10.242983  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:10.251096  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:10.255012 12171 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.256803  8579 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.214100 12864 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:10.266561  8651 raft_consensus.cc:2376] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9. Re-sending same reply.
I20260709 13:03:10.271963  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:10.277732  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63323.6us
I20260709 13:03:10.279476  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.281178 12650 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.281551 12171 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.281764  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.282614  8651 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:10.283478  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.284382  8651 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:10.290550  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51877.6us
I20260709 13:03:10.265642 12863 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.292122  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.292994  8575 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.218143 12755 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 13:03:10.295135  8728 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:10.295886  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.298014  8650 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:03:10.302234  8653 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:10.302382 12864 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.303130 12655 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.303797 12655 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.278290 12577 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.285569 12531 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:03:10.305536  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.294315 12863 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.305388 12864 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.327842  8730 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
I20260709 13:03:10.275132 12779 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.306164 12649 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.274120 12796 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.318127  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.301116 12791 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:10.337805  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:10.327936 12845 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.787s	user 0.009s	sys 0.017s
I20260709 13:03:10.338477  8650 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:10.302278 12616 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.359822 12779 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.360291  8577 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.359840 12616 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.308938  8478 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 7, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 7 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:03:10.298507 12783 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.363219 12616 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.364019  8478 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 6, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 6 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:10.366076  8733 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:10.365988  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.366873  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.367355  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.366086  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.368626  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.340816 12649 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:10.337019  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 86060.3us
I20260709 13:03:10.292284 12455 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.366120 12717 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.337920  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:10.351387  8726 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 13:03:10.351577  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.354048 12655 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.356020  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.281790 12650 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.381965  8579 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:10.382376  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:10.383515 12650 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.383649  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:10.384343  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.359154 12796 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.348559  8583 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:10.367475  8724 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:10.386190  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.308467 12615 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.370064  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.305574 12517 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:10.381425  8726 pending_rounds.cc:85] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 3
I20260709 13:03:10.397995  8726 pending_rounds.cc:107] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 13:03:10.398330  8726 raft_consensus.cc:2889] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:10.389688  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.369382  8652 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:10.371136 12649 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.394816 12531 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:10.395304 12532 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.401849  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.396276 12447 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:10.396683  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:10.402033  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:10.402130  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.401079  8577 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:10.404855  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.406929  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:10.365275 12783 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.409512  8728 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:10.411707  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.416503  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:10.430145  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.437067 12863 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.432096 12779 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:10.438542 12171 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:10.419602  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.439083 12171 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.439306  8724 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.439563 12171 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:10.420197  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.442617  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.443236  8727 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:10.443602 12863 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.444451  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.445071  8583 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.433493 12517 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.445799  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.446118  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.446425  8727 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:10.446882  8576 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:10.447022 12517 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.447296  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.447858  8727 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.407903 12531 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:10.448271  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.445222  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.449261  8578 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:10.449205 12531 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.434010  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:10.419138  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.450610  8733 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:10.451607  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.422827 12862 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.451921 12649 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:10.452370  8733 raft_consensus.cc:2410] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:03:10.421774  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.453187  8725 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:10.453456  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.420703  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.455112  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.457275  8725 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:10.434917  8478 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:10.457851  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:10.437093 12455 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:10.459370  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.426136 12486 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:10.460281 12486 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:10.422564 12853 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.460695  8655 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.417608 12615 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:10.431205  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.434509 12655 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:10.461818  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.461688 12615 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.462189 12655 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:10.462931 12655 consensus_queue.cc:1243] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:10.463205 12665 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:10.463485 12655 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:10.463609 12665 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:10.463944 12655 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:10.464155 12615 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.464623 12665 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.445799  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.452401 12649 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.466413 12649 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:10.462936  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.466935  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.467867  8655 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:10.468019  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.468560  8659 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:03:10.461009 12853 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.435065  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.459208  8478 catalog_manager.cc:5697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 6, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 6 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:10.422576 12796 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:10.470014 12796 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.470451  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:10.470959  8656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.146s	user 0.029s	sys 0.026s
I20260709 13:03:10.471035 12853 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.465966  8581 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.429699 12547 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:10.472164 12547 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.472404 12796 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.422498  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.474246 12547 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:10.474417  8726 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:10.474911  8658 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.475529  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:10.472291  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08003e90c0 after lost signal to thread 8608
I20260709 13:03:10.476188  8726 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:10.476265  8575 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
W20260709 13:03:10.476516  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08001d1500 after lost signal to thread 8609
I20260709 13:03:10.476958  8579 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:10.477056  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:10.477710  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08000ddb20 after lost signal to thread 8683
I20260709 13:03:10.477929  8726 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:10.416600  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:10.478874  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08003e19a0 after lost signal to thread 8757
I20260709 13:03:10.476210  8658 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:03:10.479467  8659 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:10.480041  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08003c6120 after lost signal to thread 9355
W20260709 13:03:10.480340  8513 debug-util.cc:398] Leaking SignalData structure 0x7b0800237ae0 after lost signal to thread 9356
I20260709 13:03:10.480476  8658 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:10.479120  8650 raft_consensus.cc:2410] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:03:10.424101  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.482029  8657 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:10.482488  8478 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 4, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 4 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:10.484356  8478 catalog_manager.cc:5697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 5, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 5 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:10.464951 12655 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.486027 12655 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.473302  8729 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:10.471215  8728 raft_consensus.cc:2410] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:03:10.489499 12447 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:10.469213  8655 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:10.434685 12354 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:10.456107  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.490640  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:10.489195  8581 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:10.491374  8578 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:10.491873  8725 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:10.494156  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:10.495074  8513 debug-util.cc:398] Leaking SignalData structure 0x7b080034fdc0 after lost signal to thread 12724
I20260709 13:03:10.429041  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:10.517225  8733 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
W20260709 13:03:10.521649  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 309611us
W20260709 13:03:10.536487  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 46047.4us
I20260709 13:03:10.460732 12486 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.496807  8583 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:10.542387  8656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.217s	user 0.029s	sys 0.027s
I20260709 13:03:10.547523  9328 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:10.547631  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:10.497445 12447 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.551954  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.545540  8478 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 5, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 5 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:03:10.516948 12290 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:10.549386  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.552788  8583 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.553299 12290 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.553746 12290 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.556447  8727 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:10.561331  8655 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:10.560559  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:10.544945 12497 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.572641  8625 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:10.545238 12859 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:10.573117  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.547186 12616 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:10.574311  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.570667 12354 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 13:03:10.577184 12514 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 13:03:10.578125 12532 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:10.581162 12517 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:10.581692 12664 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 13:03:10.583415  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:10.584128 12455 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:10.563926 12665 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.575781 12859 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.592444  8578 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:10.620786  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.630419  8478 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:10.633129  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:10.628834  8623 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.113s (SENT) [suppressed 38 similar messages]
I20260709 13:03:10.634133  8583 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:10.637686  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54339.4us
I20260709 13:03:10.638434  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:10.639051  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59008.9us
I20260709 13:03:10.640640  8652 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:10.643921  8478 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 6, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 6 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:03:10.645697  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84166.8us
I20260709 13:03:10.647467  8478 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 8, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 8 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:03:10.651186  8478 catalog_manager.cc:5697] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 6, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 6 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:03:10.598394 12517 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:10.660444 12514 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:10.678539  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:10.633910 12859 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.584144 12714 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:03:10.632822 12268 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:10.620532 12497 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:10.742053  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63274.3us
I20260709 13:03:10.717046  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:10.754925  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 188539us
I20260709 13:03:10.683280 12635 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:10.781445  8478 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:10.804809  9330 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:10.811517  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.812225  8578 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:10.813476  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.817374  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.821023  8724 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:10.681887 12514 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:10.824450  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 177829us
I20260709 13:03:10.788326 12635 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.828544  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2470) took 2742 ms. Trace:
I20260709 13:03:10.795871 12268 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:10.828946  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 85246.4us
W20260709 13:03:10.831882  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.831740  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.837067  8578 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:10.837141  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 208321us
W20260709 13:03:10.845747  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56995.9us
I20260709 13:03:10.809914 12290 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.830107 12717 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:10.846709  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.831384 12877 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.834878 12635 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:10.860512  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:10.865053 12290 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.865518  8580 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.866328  8730 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:10.869175 12715 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:03:10.869959 12881 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:03:10.836459 12655 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 13:03:10.834488 12517 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.874603 12517 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:10.875764  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.876557  8581 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:10.876204 12514 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.877592 12514 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:10.879132 12717 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.830417 12889 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 13:03:10.880594 12889 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:10.880970 12889 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:10.850091 12268 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.881510 12877 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.881999 12163 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:10.882404 12163 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:10.883239 12163 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:10.884536 12163 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:10.886168 12497 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:10.885735 12649 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.889496  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:10.889793  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:10.890005 12883 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 13:03:10.890925 12884 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:03:10.891604  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.927876  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.891911  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.869686 12715 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.929105  8581 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:10.929540 12715 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:10.893509  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:10.931744 12892 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:03:10.932300  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.932936  8583 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.859992 12547 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:10.933362  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.894060  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:10.894249  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 232816us
I20260709 13:03:10.894822  8578 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:10.935002  8576 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.881538 12888 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 13:03:10.895835  8579 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:03:10.931792  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.896477  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:10.898890  8696 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.525s (SENT)
I20260709 13:03:10.899883 12171 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:10.900527  9328 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:10.832537  8656 rpcz_store.cc:276] 0709 13:03:08.069443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.075743 (+  6300us) service_pool.cc:224] Handling call
0709 13:03:08.076089 (+   346us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:08.076488 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:08.076514 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:08.110354 (+ 33840us) log.cc:844] Serialized 52 byte log entry
0709 13:03:10.561013 (+2450659us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:10.577157 (+ 16144us) log.cc:844] Serialized 14 byte log entry
0709 13:03:10.741350 (+164193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.787416 (+ 46066us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:10.811702 (+ 24286us) raft_consensus.cc:1685] finished
0709 13:03:10.811718 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.811855 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":337781,"prepare.queue_time_us":577,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":59889,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":63351,"thread_start_us":2327321,"threads_started":2,"wal-append.queue_time_us":397354,"wal-append.run_cpu_time_us":4295,"wal-append.run_wall_time_us":1597551}
I20260709 13:03:10.872972  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:10.883626 12877 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:10.905969  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45515.5us
I20260709 13:03:10.952090  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.952363  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:10.952858  8582 raft_consensus.cc:2376] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10. Re-sending same reply.
I20260709 13:03:10.954041  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.955165  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.905766 12717 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:10.962298  8730 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:10.913889  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.919037  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:10.962453  8583 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:10.920441  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:10.923656  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:10.924777 12665 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:03:10.971494  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51117.6us
I20260709 13:03:10.925098  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:10.936834  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:10.972761  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:10.971529  9326 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.945758  8731 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:10.961653  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.934160  8577 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:10.936239  8584 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
W20260709 13:03:10.963212  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 90305.2us
W20260709 13:03:10.950282  9328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 989) took 1630 ms (client timeout 2113 ms). Trace:
W20260709 13:03:10.905742  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46003.8us
W20260709 13:03:10.976646  9328 rpcz_store.cc:269] 0709 13:03:09.319545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.428643 (+1109098us) service_pool.cc:224] Handling call
0709 13:03:10.950201 (+521558us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:10.980764  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:10.981880  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46680.4us
I20260709 13:03:10.982275  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:10.982472  8728 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:10.949156 12497 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:10.988296  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:10.859432 12664 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 13:03:10.990645  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:10.990908  8726 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:10.991586  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:10.956773 12893 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 13:03:11.002444  8583 raft_consensus.cc:2368] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:10.959332 12655 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:10.924777 12709 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:03:11.020433  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:11.020633  8578 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:03:11.021008  8584 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:11.025151  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 72872.6us
I20260709 13:03:10.987720 12497 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:10.996853  8580 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:10.980319 12791 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:11.031481  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:10.988622 12665 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:10.987232 12615 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:10.980611 12547 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:11.077677  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.077970  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:11.033589  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.984354 12796 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:11.046204  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.047039  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.049420 12709 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:11.036592 12889 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:11.052520 12715 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.069825  8581 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:11.085276  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.085975  8583 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [5.5-8.6]
I20260709 13:03:11.086208  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.086352  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:11.086898  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.070997  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:11.071413  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:11.029699 12342 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.078351  9322 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:11.025178  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:11.080116  8584 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 13:03:11.033859 12577 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:03:11.093150  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 194270us
I20260709 13:03:11.061753 12714 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:11.065554 12497 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:11.088066  8580 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:11.040277 12655 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:11.099716  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:11.100973  8581 raft_consensus.cc:2368] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9.
I20260709 13:03:11.091781  8577 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [9.6-10.7]
W20260709 13:03:11.084578  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 93999.2us
I20260709 13:03:11.082779  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:10.999396 12888 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.079s
I20260709 13:03:11.072777 12665 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.066113 12791 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.078635 12615 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.115206  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.125039  9324 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
I20260709 13:03:11.102439  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.069466 12447 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.102682  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:11.085171 12268 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:11.110875  9331 raft_consensus.cc:2376] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8. Re-sending same reply.
I20260709 13:03:11.081581 12796 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:11.081854 12709 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:11.117148  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:11.090798 12664 consensus_queue.cc:1243] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 13:03:11.143500  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 152859us
I20260709 13:03:11.155494  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.119530  9323 raft_consensus.cc:2368] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8.
I20260709 13:03:11.092739 12616 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:11.123425  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:11.093186 12577 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:11.099830  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:11.151051  9322 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:11.125909 12665 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:11.158586  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.125032 12889 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:11.166491  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.167110  8579 raft_consensus.cc:2368] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9.
I20260709 13:03:11.095870 12342 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.175930  9327 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:11.171712  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:11.173803  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 74044.9us
I20260709 13:03:11.126879 12714 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:11.131388 12615 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:11.125566 12791 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:11.185039  8576 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:11.186476  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:11.179147 12629 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:11.151227 12796 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:11.173467 12616 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:11.050695 12899 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
W20260709 13:03:11.193665  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 99548.4us
I20260709 13:03:11.200143  8657 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:11.187980 12895 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:11.136003 12650 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 13:03:11.176918 12577 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:11.197970 12532 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:11.174564 12649 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:11.202837  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:11.228403  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 197058us
W20260709 13:03:11.229846  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48999.4us
I20260709 13:03:11.200523  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.234416  8575 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:11.191998  8625 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:11.164650 12531 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
W20260709 13:03:11.248598  8696 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.598s (SENT)
I20260709 13:03:11.233958 12616 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:11.265872  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 73936.1us
W20260709 13:03:11.247542 12486 consensus_meta.cc:287] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.000s
I20260709 13:03:11.271759  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:11.272104 12486 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:11.272683  8696 leader_election.cc:336] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.622s (SENT)
I20260709 13:03:11.274384 12486 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:11.273211  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:11.276768  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:11.278704  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.279417  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.279924  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.241187 12532 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.201202 12635 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.245440 12888 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.331503  8575 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
W20260709 13:03:11.332306  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 129521us
I20260709 13:03:11.344656  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.284014  8733 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:11.335717  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.349561  9331 raft_consensus.cc:2376] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 9. Re-sending same reply.
I20260709 13:03:11.337613  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.341312  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.343931  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.347817  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:11.351974  9330 raft_consensus.cc:2410] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 13:03:11.194993 12547 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.233660 12709 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.359534 12709 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:11.229342 12655 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:11.279771 12629 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:03:11.355255  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.355528  9325 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:11.337071 12354 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:11.358336  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:11.333771  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.400174  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 127527us
I20260709 13:03:11.407243  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.408274  9324 raft_consensus.cc:2368] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9.
I20260709 13:03:11.344859 12715 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:11.410313  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.423328  9331 raft_consensus.cc:2376] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 9. Re-sending same reply.
I20260709 13:03:11.338059 12532 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:11.419950  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.401023  8696 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.666s (SENT)
I20260709 13:03:11.456231  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:11.444366  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.468371  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.366711 12717 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:11.477754  8576 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:11.479609  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.503679  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.348776 12888 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.512933  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.518229  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:11.483245 12871 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 13:03:11.483291 12897 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260709 13:03:11.548900  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.535634  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:11.481966 12895 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:11.481745 12717 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:11.557688  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:11.510314 12906 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 13:03:11.483196 12910 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:03:11.566490  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.566365  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:11.500233 12447 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:11.574512  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:11.510298 12898 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:11.514099 12914 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:11.624579  8576 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
W20260709 13:03:11.678567  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 406840us
I20260709 13:03:11.679853  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:11.684490  8579 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:11.686327  8579 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:11.688916  9328 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:11.399938 12714 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.708432  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:11.563275 12717 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:11.730343  8576 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 13:03:11.734851  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 387145us
W20260709 13:03:11.739363  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.760396  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.777051  9326 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.797s	user 0.017s	sys 0.000s
I20260709 13:03:11.777618  9326 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:11.621843 12910 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:11.582741 12486 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:11.784457  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 383416us
W20260709 13:03:11.834511 12904 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.549s	user 0.001s	sys 0.000s
W20260709 13:03:11.806955 12871 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:03:11.851159  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.798223 12910 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:11.801257  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:11.813337  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:11.800117 12918 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:11.682192 12914 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:11.855924  8697 leader_election.cc:400] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9.
W20260709 13:03:11.834168 12903 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.581s	user 0.001s	sys 0.001s
I20260709 13:03:11.548386 12917 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.224s
I20260709 13:03:11.809759  8581 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:11.911082  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:11.911712  8581 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:03:11.819298  8576 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
W20260709 13:03:11.876246  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 196495us
I20260709 13:03:11.914743  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:11.800647 12455 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:03:11.879904  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:11.880761  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.001s	sys 0.009s
I20260709 13:03:11.882427  9325 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 13:03:11.921687  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.001s	sys 0.009s
W20260709 13:03:11.885272  8696 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.724s (SENT)
W20260709 13:03:11.922108  8582 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.828s	user 0.015s	sys 0.004s
W20260709 13:03:11.886181  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.887179  8622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.123s	sys 0.179s
W20260709 13:03:11.887598  9327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.002s	sys 0.014s
W20260709 13:03:11.923216  9327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.003s	sys 0.015s
W20260709 13:03:11.882087 12531 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.026s	sys 0.064s
I20260709 13:03:11.893355  8697 leader_election.cc:403] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260709 13:03:11.926889  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 71095.7us
W20260709 13:03:11.927259  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47452.7us
I20260709 13:03:11.927832  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:11.928503  8548 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:11.894117  8478 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.006s	sys 0.011s
I20260709 13:03:11.893143  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 800) took 1004 ms. Trace:
I20260709 13:03:11.929328  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:11.876226 12635 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:11.917142  8657 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.676s	user 0.015s	sys 0.014s
I20260709 13:03:11.932245  8657 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:03:11.922282  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 850791us
W20260709 13:03:11.885844  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.889699 12342 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:11.923102  8622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.128s	sys 0.179s
I20260709 13:03:11.924789  8582 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:11.938162  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2622) took 1043 ms. Trace:
I20260709 13:03:11.929334  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:11.896670 12918 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:11.931094  8478 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.875s	user 0.007s	sys 0.011s
W20260709 13:03:11.912135 12859 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.125s	user 0.011s	sys 0.010s
I20260709 13:03:11.933873  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:11.930090 12531 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.026s	sys 0.065s
I20260709 13:03:11.938508  8582 rpcz_store.cc:276] 0709 13:03:10.895212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.981098 (+ 85886us) service_pool.cc:224] Handling call
0709 13:03:11.938101 (+957003us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25984}
W20260709 13:03:11.945333  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 67334.3us
W20260709 13:03:11.946171 12924 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 13:03:11.896533 12914 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:11.947961 12923 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:03:11.943338  8575 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 13:03:11.946110  8696 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.761s (SENT)
I20260709 13:03:11.949265  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:11.949626 12926 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:11.950089 12926 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.950420 12926 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:11.950474  8696 leader_election.cc:336] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.800s (SENT)
I20260709 13:03:11.951364 12925 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.940621  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:11.952803 12514 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.075s	user 0.010s	sys 0.000s
W20260709 13:03:11.954083 12915 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 13:03:11.941820  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 665081us
W20260709 13:03:11.955698 12933 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 13:03:11.957039 12920 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.373s	user 0.001s	sys 0.000s
I20260709 13:03:11.929874  8729 rpcz_store.cc:276] 0709 13:03:10.888860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.892098 (+  3238us) service_pool.cc:224] Handling call
0709 13:03:10.892412 (+   314us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:10.892904 (+   492us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:10.892932 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:10.893567 (+   635us) log.cc:844] Serialized 52 byte log entry
0709 13:03:10.970205 (+ 76638us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:10.970275 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.970341 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:11.892910 (+922569us) raft_consensus.cc:1685] finished
0709 13:03:11.892930 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:11.893082 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":64222,"threads_started":1,"wal-append.queue_time_us":233696}
I20260709 13:03:11.959250  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2632) took 1011 ms. Trace:
I20260709 13:03:11.893512 12486 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.973264  8581 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:11.973194  8578 rpcz_store.cc:276] 0709 13:03:10.948337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.039164 (+ 90827us) service_pool.cc:224] Handling call
0709 13:03:11.039613 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:11.477835 (+438222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:11.477866 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:11.503926 (+ 26060us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.866327 (+362401us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:11.866364 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.866437 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:11.958989 (+ 92552us) raft_consensus.cc:1685] finished
0709 13:03:11.959015 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:11.959179 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3584,"raft.queue_time_us":298195,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":28131,"thread_start_us":644661,"threads_started":2,"wal-append.queue_time_us":274029}
I20260709 13:03:11.973907  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:11.934945 12354 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9.
I20260709 13:03:11.935443 12650 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:11.945216 12935 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:11.980846  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:11.965842 12163 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.040s	user 0.008s	sys 0.004s
I20260709 13:03:11.944744 12918 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:11.942988 12447 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:11.985028  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:11.964876 12342 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.966391 12514 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:11.979739 12889 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.790s	user 0.014s	sys 0.000s
I20260709 13:03:11.960520 12931 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:11.988392  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:11.989959  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.977272 12531 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:03:12.003041  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 189729us
I20260709 13:03:12.005119  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 795) took 1118 ms. Trace:
W20260709 13:03:12.006963  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 92452.4us
I20260709 13:03:12.006162  8727 rpcz_store.cc:276] 0709 13:03:10.887140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.887271 (+   131us) service_pool.cc:224] Handling call
0709 13:03:10.887654 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:10.888137 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:10.888169 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:10.888674 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.084569 (+195895us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:11.084619 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.084686 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.004775 (+920089us) raft_consensus.cc:1685] finished
0709 13:03:12.004810 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.005031 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2049,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":195502,"threads_started":1,"wal-append.queue_time_us":286789}
W20260709 13:03:12.007710  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53426.3us
I20260709 13:03:12.007948  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:12.008379  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:12.008483  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 67921.9us
W20260709 13:03:12.010133  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 76344.4us
I20260709 13:03:12.012326  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:12.016144  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:11.983748 12935 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:11.962764 12859 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:11.985882 12888 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.003592 12931 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:12.032648  8548 leader_election.cc:400] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8.
I20260709 13:03:12.033036  8548 leader_election.cc:403] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:12.035404 12943 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:12.035873 12943 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.036211 12943 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:12.036921  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48615.2us
I20260709 13:03:12.019201 12937 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:12.037979  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:12.035981 12486 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:12.039347  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 110115us
I20260709 13:03:12.048750  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:12.049862 12888 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.052443  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:12.035420  8696 leader_election.cc:400] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:12.026882 12935 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:12.005219 12342 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.056452  8696 leader_election.cc:403] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:03:12.054817 12945 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:12.057346 12913 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:12.058161  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50319.5us
I20260709 13:03:12.059088 12930 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:12.037648 12919 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.224s	user 0.002s	sys 0.000s
I20260709 13:03:12.031837 12931 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:12.061336 12497 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.946s	user 0.010s	sys 0.002s
I20260709 13:03:12.075327 12945 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.951867 12925 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:12.135332  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 128944us
I20260709 13:03:12.135928 12952 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:03:12.136829  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59364.4us
I20260709 13:03:12.081003 12917 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.125777 12913 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
W20260709 13:03:12.123463  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 92314us
W20260709 13:03:12.137797  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 157008us
I20260709 13:03:12.076066 12940 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:12.148116  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:12.148358  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110489us
W20260709 13:03:12.148923  8697 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4f350f118fc449fb61357528a6b14f7. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:03:12.111055 12447 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:12.158169 12447 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.159302  8650 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:12.160879  8656 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260709 13:03:12.162206  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:12.148164  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 112866us
I20260709 13:03:12.149705  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:12.163820  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 151534us
I20260709 13:03:12.164638  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:12.166504  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:12.111168 12497 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:12.170375 12927 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
W20260709 13:03:12.173712 12791 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.935s	user 0.006s	sys 0.005s
I20260709 13:03:12.174141 12791 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.175711  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:12.176254 12791 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.174500 12342 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.146381 12945 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:12.146955 12928 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:12.180500  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:12.080473 12930 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.188535  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2682) took 1030 ms. Trace:
I20260709 13:03:12.178362  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:12.148401 12531 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.153921 12952 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.194227  8652 rpcz_store.cc:276] 0709 13:03:11.158239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.169263 (+ 11024us) service_pool.cc:224] Handling call
0709 13:03:11.198638 (+ 29375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:11.234754 (+ 36116us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:11.234778 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:11.264339 (+ 29561us) log.cc:844] Serialized 492 byte log entry
0709 13:03:11.391972 (+127633us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:11.392048 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.392166 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.188229 (+796063us) raft_consensus.cc:1685] finished
0709 13:03:12.188260 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.188451 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":127063,"threads_started":1,"wal-append.queue_time_us":356740}
I20260709 13:03:12.197250 12342 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:12.198432  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48818.9us
W20260709 13:03:12.190341 12517 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.315s	user 0.007s	sys 0.008s
I20260709 13:03:12.199316  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:12.202215  9331 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:12.204470 12922 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 13:03:12.189994 12940 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.194286 12928 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:12.199893 12889 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.196666 12791 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:12.198127 12486 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.198061 12354 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.196918 12937 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.201762 12342 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:12.205587 12888 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:12.206210 12951 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:12.220602 12517 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.223398 12859 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.219281 12939 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:12.222318 12928 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:12.221220 12949 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:12.214148 12913 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:03:12.235603 12942 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.244879 12342 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:12.244799 12791 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.268200 12918 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:12.270773 12268 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.119s	user 0.010s	sys 0.011s
I20260709 13:03:12.292516 12268 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:12.293321 12949 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:12.275426  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 991) took 1024 ms. Trace:
I20260709 13:03:12.230242 12889 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.277999 12664 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:03:12.279158 12616 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.952s	user 0.007s	sys 0.003s
I20260709 13:03:12.260842 12939 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.247963 12921 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:12.236217 12486 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.293443  8651 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:12.235984 12354 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:12.293195 12947 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 13:03:12.241788 12944 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.308324 12950 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.273056 12717 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.308825 12930 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:12.302832  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 147021us
I20260709 13:03:12.293260 12951 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 13:03:12.294644  9329 rpcz_store.cc:276] 0709 13:03:11.251592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.251745 (+   153us) service_pool.cc:224] Handling call
0709 13:03:11.279609 (+ 27864us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:11.334837 (+ 55228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:11.334872 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:11.340074 (+  5202us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.543110 (+203036us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:11.543169 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.543265 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.275113 (+731848us) raft_consensus.cc:1685] finished
0709 13:03:12.275141 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.275340 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3070,"raft.queue_time_us":43462,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4915,"thread_start_us":202528,"threads_started":1,"wal-append.queue_time_us":342301}
I20260709 13:03:12.306586 12948 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 7. Current term is 8.
I20260709 13:03:12.307579 12921 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.307886 12959 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:12.307941 12486 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:12.321818  8730 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:03:12.322854  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 123611us
I20260709 13:03:12.300037 12664 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:12.328625  8727 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:12.329064  8696 leader_election.cc:400] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9.
I20260709 13:03:12.329762  8731 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:12.304955 12940 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.331668  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:12.332260  8724 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:03:12.333540 12950 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:12.333942 12950 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:12.334107 12925 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.330120  8731 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:12.337738  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:12.319849 12532 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.806s	user 0.011s	sys 0.000s
I20260709 13:03:12.341933  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:12.363114  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:12.364051  8732 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:03:12.370189  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 151329us
I20260709 13:03:12.304941 12616 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:12.375358  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 199700us
I20260709 13:03:12.333972 12791 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.298318 12531 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.387882  8724 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:12.361948  8731 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.345664 12930 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:12.396075  8733 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:12.371297 12957 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.398041  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.399051  8657 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:12.282588 12952 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.396637  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.401432  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.402304  8731 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:12.409438  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.409660  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.410077  8656 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:12.412673  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.348129 12532 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.416476  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.419366  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.422528  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.422976  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:12.423326 12171 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.423767  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:12.424042  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:12.424405  8653 raft_consensus.cc:2376] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 8. Re-sending same reply.
I20260709 13:03:12.387156 12859 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:12.425379  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.410658  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.424768  8656 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:12.431857  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.432055  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.436426  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:12.437826 12964 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.438563 12964 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.438992  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.439733  8658 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:12.440327 12859 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.440961 12964 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.441668  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.442549 12616 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.437753  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:12.325855 12937 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.443966  8653 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:12.451906  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:12.458628  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.459530  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:12.345125 12950 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.407064 12930 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.415741 12939 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.471912  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.457744  8731 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 13:03:12.457175 12577 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.198s	user 0.010s	sys 0.003s
I20260709 13:03:12.459028 12964 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 13:03:12.481547 12958 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.459208 12888 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:12.482463 12958 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:12.484728 12908 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.047s	user 0.000s	sys 0.004s
I20260709 13:03:12.484218 12888 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.458562 12959 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:12.407809 12957 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.489030  8655 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:12.458155 12717 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.500226  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:12.513820  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 61966.6us
I20260709 13:03:12.515021  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:12.516712  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:12.520730  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:12.576460  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55820.8us
I20260709 13:03:12.460722 12171 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 13:03:12.428669 12664 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.494022 12959 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:12.605144  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 364460us
I20260709 13:03:12.487720 12889 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:12.588505  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:12.587030  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 198336us
I20260709 13:03:12.589313  8654 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:12.487289 12918 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:12.522760 12709 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.163s	user 0.016s	sys 0.018s
W20260709 13:03:12.563102 12665 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.359s	user 0.013s	sys 0.009s
I20260709 13:03:12.594739 12664 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:12.596163 12968 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:12.523059 12354 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:12.609154  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:12.609746  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:12.600478 12931 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:12.619961  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:12.429281 12921 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.580906 12944 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:12.616740  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:12.688419  8727 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:12.488056 12577 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.592490 12925 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:12.685958  8622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.050s	sys 0.085s
I20260709 13:03:12.491552 12928 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.609494 12959 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:12.614930 12968 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.679934 12709 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:12.702289  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2644) took 1650 ms. Trace:
I20260709 13:03:12.702744  8575 rpcz_store.cc:276] 0709 13:03:11.052382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.342390 (+290008us) service_pool.cc:224] Handling call
0709 13:03:11.939006 (+596616us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:11.948878 (+  9872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.702077 (+753199us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54003a3c50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:12.702225 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320984448}
W20260709 13:03:12.693230  8622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.050s	sys 0.085s
W20260709 13:03:12.699684 12796 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.459s	user 0.012s	sys 0.008s
I20260709 13:03:12.669198 12665 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.697963 12532 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:12.704731  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:12.704668  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 540040us
W20260709 13:03:12.703401 12977 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260709 13:03:12.714666 12938 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.756s	user 0.006s	sys 0.000s
I20260709 13:03:12.714339 12342 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.702333 12709 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.583864  9329 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:12.717124  9329 raft_consensus.cc:606] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:12.717536  9329 raft_consensus.cc:614] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:12.718544  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 818) took 1437 ms. Trace:
I20260709 13:03:12.718920  8580 rpcz_store.cc:276] 0709 13:03:11.281589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.728739 (+447150us) service_pool.cc:224] Handling call
0709 13:03:11.800331 (+ 71592us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:11.809253 (+  8922us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:11.809290 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:11.830501 (+ 21211us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.920049 (+ 89548us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:11.920109 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.920219 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.713863 (+793644us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:12.718286 (+  4423us) raft_consensus.cc:1685] finished
0709 13:03:12.718307 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.718489 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":57047,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":25020,"spinlock_wait_cycles":11662080,"thread_start_us":83499,"threads_started":1,"wal-append.queue_time_us":122140}
W20260709 13:03:12.720034 12615 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.515s	user 0.009s	sys 0.012s
I20260709 13:03:12.720451 12615 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.721701  8583 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:12.709946 12796 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:12.722954 12914 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.775s	user 0.011s	sys 0.000s
I20260709 13:03:12.724006  8582 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:12.712100  8654 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:12.713433 12859 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.725992  8652 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:12.732080 12925 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:03:12.734848 12943 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.698s	user 0.013s	sys 0.001s
I20260709 13:03:12.735394 12943 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.736572 12786 raft_consensus.cc:2764] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 13:03:12.704512 12968 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:12.725970 12910 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.815s	user 0.000s	sys 0.011s
I20260709 13:03:12.708503 12944 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.746104  8725 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:03:12.749547 12982 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:12.723385 12914 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:12.751044 12955 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.490s	user 0.007s	sys 0.013s
I20260709 13:03:12.748845  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:12.756181 12615 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.726964 12958 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.724910 12342 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.757763 12709 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.757426 12981 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:03:12.761968 12896 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
W20260709 13:03:12.793641 12945 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.589s	user 0.009s	sys 0.003s
I20260709 13:03:12.794196 12945 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.796821 12930 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:12.798086 12978 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 13:03:12.791215 12916 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:12.798688 12916 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.799775 12916 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.801119 12916 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:12.802161 12957 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.805723 12629 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:12.806174 12629 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:12.806984 12943 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.807083 12629 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.769120 12944 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.754053 12917 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 13:03:12.765753 12910 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.810745 12547 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.811304  8579 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:12.811487 12547 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.812573  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 817) took 1555 ms. Trace:
I20260709 13:03:12.812983  8579 rpcz_store.cc:276] 0709 13:03:11.256908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.704112 (+447204us) service_pool.cc:224] Handling call
0709 13:03:11.704443 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:12.811737 (+1107294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.812339 (+   602us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540002dd54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:12.812511 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2544841088}
I20260709 13:03:12.814519  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2685) took 1653 ms. Trace:
I20260709 13:03:12.815246  8651 rpcz_store.cc:276] 0709 13:03:11.160912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.169568 (+  8656us) service_pool.cc:224] Handling call
0709 13:03:11.194004 (+ 24436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:12.310362 (+1116358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.814216 (+503854us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400380254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:12.814443 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2527730560}
I20260709 13:03:12.816108 12928 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.816475 12937 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:12.817296 12918 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:12.817878 12918 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.819984 12940 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:12.820708 12940 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.823060 12940 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.823201  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.824007 12937 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.824093  8650 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 5.
W20260709 13:03:12.824820 12935 consensus_meta.cc:287] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.749s	user 0.010s	sys 0.000s
I20260709 13:03:12.814628 12665 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:12.815656  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66888.7us
I20260709 13:03:12.808482 12629 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:12.817124  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:12.820065 12918 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.845283  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:12.850637  8650 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:12.849807 12928 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:12.853096 12928 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.857039  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 824) took 1056 ms. Trace:
I20260709 13:03:12.864346  8651 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 13:03:12.827890 12944 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.837252 12655 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:12.830256 12914 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.825309 12935 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:12.868393 12935 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.857472  8576 rpcz_store.cc:276] 0709 13:03:11.800946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.916411 (+115465us) service_pool.cc:224] Handling call
0709 13:03:11.937304 (+ 20893us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:11.940100 (+  2796us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:11.957695 (+ 17595us) log.cc:844] Serialized 14 byte log entry
0709 13:03:12.775370 (+817675us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b1800374f20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:03:12.856780 (+ 81410us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:12.856823 (+    43us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:12.856865 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.856913 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.856996 (+    83us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1968,"prepare.queue_time_us":21895,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":181601,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":3314,"spinlock_wait_cycles":6116608,"thread_start_us":131976,"threads_started":2,"wal-append.queue_time_us":80926}
I20260709 13:03:12.840600 12910 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.864217 12918 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:12.889737  8651 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:12.878894 12935 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:12.825183 12937 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.896345  8659 raft_consensus.cc:2368] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
I20260709 13:03:12.896662  8651 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:12.899554  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:12.900352  8657 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:12.893020 12926 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.942s	user 0.012s	sys 0.001s
I20260709 13:03:12.912515 12926 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:12.913792 12497 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.915341 12926 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.916378  8650 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: [4.5-8.6]
I20260709 13:03:12.918210  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:12.919495  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2634) took 1920 ms. Trace:
I20260709 13:03:12.921981  8577 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:03:12.923882  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2647) took 1870 ms. Trace:
I20260709 13:03:12.924317  8577 rpcz_store.cc:276] 0709 13:03:11.062960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.687781 (+624821us) service_pool.cc:224] Handling call
0709 13:03:12.921513 (+1233732us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:12.922465 (+   952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.923496 (+  1031us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001694d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:12.923675 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2836155776}
I20260709 13:03:12.936476 12547 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:12.907910 12945 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.896365 12918 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:12.938246 12918 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.897215 12935 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.943353 12928 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.943425 12935 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.919996  8584 rpcz_store.cc:276] 0709 13:03:10.999569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.103222 (+103653us) service_pool.cc:224] Handling call
0709 13:03:11.126990 (+ 23768us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:11.238576 (+111586us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:11.238609 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:11.254842 (+ 16233us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.398996 (+144154us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:11.698787 (+299791us) log.cc:844] Serialized 14 byte log entry
0709 13:03:12.918914 (+1220127us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b540033eb20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b91733 00007f1409c16dbf 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c
0709 13:03:12.918985 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.919093 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.919151 (+    58us) raft_consensus.cc:1685] finished
0709 13:03:12.919259 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.919428 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13749,"prepare.queue_time_us":99839,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":18331,"raft.queue_time_us":757466,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":27544,"spinlock_wait_cycles":26160896,"thread_start_us":546859,"threads_started":3,"wal-append.queue_time_us":150190}
I20260709 13:03:12.947299 12853 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.947855 12853 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.948722 12959 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.966490  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:12.968179  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 359057us
I20260709 13:03:12.968214  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:12.969023  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:12.910998 12968 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:12.909323 12931 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:12.954494 12577 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:12.958739  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:12.958590  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 998) took 1683 ms. Trace:
I20260709 13:03:12.967420  9331 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:12.977541  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.005705  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:12.986639  9325 rpcz_store.cc:276] 0709 13:03:11.275388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.520588 (+245200us) service_pool.cc:224] Handling call
0709 13:03:11.623300 (+102712us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:11.921819 (+298519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.958291 (+1036472us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b5400386b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:12.958505 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":540778880}
I20260709 13:03:13.013882  9323 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:13.023082  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45655.2us
I20260709 13:03:13.025388  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.031159  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.033018  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:13.034534  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.034722  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.037012  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.037585  8655 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:13.037644  9325 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:13.006997 12951 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:13.042528  8650 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:03:13.043054  8650 pending_rounds.cc:85] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 8
I20260709 13:03:13.043411  8650 pending_rounds.cc:107] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:03:13.043725  8650 raft_consensus.cc:2889] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:13.045475  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:13.046346  8656 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:13.047564  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.048261  8657 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:13.048359 12968 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:13.048926 12989 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.070s	user 0.004s	sys 0.000s
I20260709 13:03:13.051678 12959 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:13.041774 12951 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:13.053380 12951 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:13.053721  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.042860  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.055284  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.058703  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:13.060158  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.060117  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.032961 12577 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:13.078753  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.080531  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.081866  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.081211  9323 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:13.044862 12990 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:13.087388  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:13.088021  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:13.090219  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.091931 12994 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.061688 12948 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:13.094075  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.066756 12939 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:13.071151 12982 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:13.070369 12889 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.066013 12898 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:13.065269 12497 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:13.106382 12497 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.108624 12497 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:13.110948  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.080598 12899 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:13.102341  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.036679 12931 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 13:03:13.070801 12935 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:13.071964 12354 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:13.080251 12928 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:03:13.113682  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70876.8us
I20260709 13:03:13.115190  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:13.116959  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.092736 12994 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:13.253997  9329 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:03:13.180441  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.167467  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 810) took 1351 ms. Trace:
I20260709 13:03:13.095306 12945 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:03:13.168102  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 88986.6us
I20260709 13:03:13.256332  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.149715 12796 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.154070  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.084399 12964 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:03:13.267266  8579 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:13.103183 12979 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.112313 12889 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:13.104439 12968 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:13.115850 12898 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.102294 12939 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:13.241670  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 825) took 1406 ms. Trace:
I20260709 13:03:13.279933  8582 rpcz_store.cc:276] 0709 13:03:11.834966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.946104 (+111138us) service_pool.cc:224] Handling call
0709 13:03:11.946398 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:12.724481 (+778083us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.240900 (+516419us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54003a4b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:13.241604 (+   704us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1787775744}
I20260709 13:03:13.241477  8650 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:13.242789  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:13.241890  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.281881  8576 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 13:03:13.251921  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:13.252048  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 135120us
I20260709 13:03:13.255664  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:13.283675  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.255458  8725 rpcz_store.cc:276] 0709 13:03:11.815676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.816812 (+  1136us) service_pool.cc:224] Handling call
0709 13:03:11.831212 (+ 14400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:12.753233 (+922021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.167106 (+413873us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b5400229a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:13.167384 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2099301120}
I20260709 13:03:13.279757  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:13.256579  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:13.284966  8575 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:13.285050  8584 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:13.289682 12796 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:13.290310 12796 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:13.290732 12796 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:13.262514  9323 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:13.256431 12913 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.274605  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.293992 12913 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.297637 12913 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.257215 12948 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:13.307775  8577 raft_consensus.cc:2368] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9.
I20260709 13:03:13.308487  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.260696 12928 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:13.312683  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.313814  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:13.315573  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 224253us
I20260709 13:03:13.320688  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.322992  8733 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:13.325795  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:13.323805  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.324648  8584 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:13.327569  8731 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 13:03:13.328783  8731 pending_rounds.cc:85] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:03:13.329107  8731 pending_rounds.cc:107] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 13:03:13.329469  8731 raft_consensus.cc:2889] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:13.331884  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.366979  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.367941  8725 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:13.369226  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.282172 12939 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:13.370296  8727 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:13.355897  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.371869  8730 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:13.256219 12354 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.335655  8726 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:13.377820  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:13.374399  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:13.401134  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 77407.7us
I20260709 13:03:13.321069 12974 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:13.433288  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:13.434412  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.429909  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.437956  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.440114  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.320509 12945 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:13.330256 12968 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:13.442461  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.437934  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.441689 12937 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.438530  8731 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:13.441128  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:13.444546  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70993.5us
I20260709 13:03:13.354499 12990 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:13.332041 12879 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:13.436589  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.451342  8726 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:13.492442  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.279153 12979 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:13.503345 12979 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:13.504382 13000 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:03:13.506191 12354 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.400039 12455 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 13:03:13.510452 12901 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:13.510896 12901 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:13.512213 12901 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.376132 12982 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:13.398087 12517 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.514034 12517 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:13.516127 12517 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:13.518356 12994 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:13.314802 12888 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.123s
I20260709 13:03:13.331925 12497 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.105s
I20260709 13:03:13.520124 12497 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:13.520488 12497 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:13.521391 12497 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:13.522066  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 85666us
I20260709 13:03:13.522666 12497 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:13.522828  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.524108  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.330698 12547 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 6, Time since last communication: 0.102s
I20260709 13:03:13.526355 12899 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:13.526729 12899 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:13.527262 12889 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:13.527634 12899 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.528578  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2500) took 1106 ms. Trace:
I20260709 13:03:13.529141  8652 rpcz_store.cc:276] 0709 13:03:12.422443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.432899 (+ 10456us) service_pool.cc:224] Handling call
0709 13:03:13.528327 (+1095428us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b540007dad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:13.528502 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":649471872}
I20260709 13:03:13.493805  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:13.495929 13006 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 13:03:13.492257 12879 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:13.443070 12665 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:13.510792 12898 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:13.498695 12968 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:13.434964 12514 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:13.490435  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:13.562397  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:13.562860  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:13.533502  8696 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.577s (SENT)
I20260709 13:03:13.452838  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.533861  8730 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:13.525288  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:13.535514  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 209831us
I20260709 13:03:13.535784  8725 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:13.541601  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.550488  8656 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
W20260709 13:03:13.577009  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:13.578178  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44704us
I20260709 13:03:13.578572  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:13.579138  8696 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.614s (SENT)
I20260709 13:03:13.579713  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.580283  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:13.582662  8730 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:13.551563  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.517908 12928 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:13.444995 12969 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:13.559875  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.583519  8733 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:13.585529  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.539755 12959 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:13.540767 12982 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:13.566102  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.554682 12879 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:13.588548  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:13.616437  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:13.618793  8625 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:13.593680  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.452289 12937 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:13.607538  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82270.7us
I20260709 13:03:13.622264  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:13.609669  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 43674.9us
I20260709 13:03:13.539992 12935 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 13:03:13.565721 12913 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.553622 12951 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.441582 12974 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.625140  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:13.573875 12994 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:13.591104 12964 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:13.554299 12665 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:13.621070 12577 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.544s	user 0.009s	sys 0.001s
I20260709 13:03:13.611022 12990 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:13.633949 12268 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:13.699470 12974 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.575908 12514 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.576982 12930 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 13:03:13.589164 12916 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:13.672931  8727 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:13.604898 12926 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:13.621392 12629 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:13.575559 12898 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 13:03:13.625219 12171 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:13.681435  8724 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:13.654064 12913 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:13.682024  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45423.2us
W20260709 13:03:13.683272  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 104167us
W20260709 13:03:13.765722  8696 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.620s (SENT)
W20260709 13:03:13.684185  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62029.4us
I20260709 13:03:13.627085  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.714322 12974 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:13.766923  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 139950us
I20260709 13:03:13.767405  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:13.612769 12969 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.660107 12982 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:03:13.668774 12952 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:03:13.672546 13007 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:03:13.634259 12547 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:03:13.702299 12577 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:03:13.728929 13010 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:03:13.733213 12853 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:13.817165  8731 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:13.774976 12950 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:13.738897 12665 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:13.744750  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 479253us
I20260709 13:03:13.785355 12982 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:13.750765 12973 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index -1 to 7, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
W20260709 13:03:13.753923 13009 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 13:03:13.674237 13012 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:03:13.708921 12916 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:13.709167 12926 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:13.710013 12629 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:13.812014 12994 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:13.656797 12966 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:13.768997  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:13.773656  8727 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:03:13.727520 12937 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:13.816111  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.768436 12951 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.734867 12514 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.777637 12898 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:13.785862 12952 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:13.891764 12952 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:13.892549 12901 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:13.821991  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:13.746834  8583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2641) took 2695 ms (client timeout 2139 ms). Trace:
W20260709 13:03:13.823088  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55734.1us
W20260709 13:03:13.896951  8583 rpcz_store.cc:269] 0709 13:03:11.051415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.086720 (+ 35305us) service_pool.cc:224] Handling call
0709 13:03:13.746556 (+2659836us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54003e7454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:13.746734 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3646725888}
I20260709 13:03:13.897960  8549 leader_election.cc:400] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
I20260709 13:03:13.898361  8549 leader_election.cc:403] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:03:13.899494 12948 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
I20260709 13:03:13.900008 13012 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:13.788612 12974 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:13.845060 12999 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.491s	user 0.001s	sys 0.000s
I20260709 13:03:13.786401 12888 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:13.767519  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:13.916292  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 148818us
I20260709 13:03:13.871923  8731 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:13.873150  8726 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:13.925851  8726 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:13.926133  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:13.874221 12926 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:13.874527 12629 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:13.901065 12898 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:13.878628  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.931833  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:13.882484  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 113584us
I20260709 13:03:13.934268  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:13.882018  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2533) took 1268 ms. Trace:
I20260709 13:03:13.872741 12982 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:13.871686 12497 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:13.884902 12532 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:13.908738  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.909950  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:13.821879 12171 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:13.926074  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.951702  9323 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:13.874768 12796 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.583s	user 0.011s	sys 0.008s
I20260709 13:03:13.926995  8726 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:13.873808 12916 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:13.927618  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.956748  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:13.957001  9331 raft_consensus.cc:2410] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:13.930172  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.957728  9329 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:13.928555 13012 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:13.961966  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.962272  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.929944  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:13.963383  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:13.939247  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.935290  8732 rpcz_store.cc:276] 0709 13:03:12.613588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.613724 (+   136us) service_pool.cc:224] Handling call
0709 13:03:12.677601 (+ 63877us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:13.168745 (+491144us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.168790 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:13.240115 (+ 71325us) log.cc:844] Serialized 26 byte log entry
0709 13:03:13.557317 (+317202us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:13.557360 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.557468 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:13.879549 (+322081us) raft_consensus.cc:1685] finished
0709 13:03:13.879577 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:13.881958 (+  2381us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":506093,"raft.queue_time_us":550732,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":19089,"thread_start_us":278268,"threads_started":1,"wal-append.queue_time_us":434022}
I20260709 13:03:13.914839 12939 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.927690 12268 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:13.921243 12650 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:13.930176  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:13.958969  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.959619  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.998744  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:13.960212  8729 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:13.937734 12898 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:13.912117  8549 connection.cc:256] RPC call timeout handler was delayed by -1.2249951259999534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260709 13:03:13.962608  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:13.973747 12993 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.733s	user 0.002s	sys 0.001s
I20260709 13:03:13.974399 12879 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.974617 12171 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.003000 12975 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:03:14.032119 12896 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:14.033097  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:13.965574 12937 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.033651  8729 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:14.033746  8582 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:14.034384 13023 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:03:14.034932  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.035521  8582 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:14.036230  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.036433  8727 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:03:14.037305  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.037918  8580 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:14.038116  8727 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:03:14.039268 12974 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:14.039829  8727 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 13:03:14.040108  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 673) took 1601 ms. Trace:
I20260709 13:03:14.040294  8727 pending_rounds.cc:85] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:03:14.040689  8727 pending_rounds.cc:107] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 13:03:13.975252 12969 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.975644 13012 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:14.042081  8655 raft_consensus.cc:2368] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:14.042948  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.042618 13012 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.043687  8582 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.044564  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.045109  8582 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:13.976675  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:13.977707  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:13.978113 12996 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.720s	user 0.000s	sys 0.002s
I20260709 13:03:13.978161 12853 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:13.982313 12952 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:13.941924 12963 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:13.941522 12497 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:14.055956 12497 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.057242 12497 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:13.984292 12916 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.060359 12916 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.985682  8724 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:13.969820 12532 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:14.061707 13024 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.062196 12532 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.062654  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:14.062407 13024 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.063226  8724 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:03:14.040635 12939 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.985260 12913 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:13.991094 12649 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:14.033794 12937 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:13.994513  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:13.994560  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:14.001528  8733 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:14.002771  8656 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:14.067996  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:14.068181 12937 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:14.068750  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:14.069139  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:14.069887  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:14.070466  8656 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:14.070616  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 670) took 1645 ms. Trace:
I20260709 13:03:14.072300  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.073531  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.074787  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.075824 12889 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:14.076035  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.076431 12889 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:14.089262  8657 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:14.099864 13008 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.429s	user 0.001s	sys 0.001s
I20260709 13:03:14.101804  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:14.103808 12948 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:14.104416 12948 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:14.104441  8696 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.722s (SENT)
I20260709 13:03:14.105973  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:14.106751 12649 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:14.114015  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2692) took 1046 ms. Trace:
I20260709 13:03:14.114598  8655 rpcz_store.cc:276] 0709 13:03:13.068161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.079935 (+ 11774us) service_pool.cc:224] Handling call
0709 13:03:14.113704 (+1033769us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54000c85d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:14.113927 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2135827712}
I20260709 13:03:14.116832  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1024) took 1043 ms. Trace:
I20260709 13:03:14.117372  9325 rpcz_store.cc:276] 0709 13:03:13.074389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.080161 (+  5772us) service_pool.cc:224] Handling call
0709 13:03:13.097184 (+ 17023us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:13.112524 (+ 15340us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.112553 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:13.112591 (+    38us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:13.112628 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.116514 (+1003886us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5400519354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:14.116578 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.116744 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34806656}
I20260709 13:03:14.123483  9329 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:14.006943  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:14.018457  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:13.967456 13018 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:14.009110  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.018074  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.133590  8578 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:14.134652  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.135434  8578 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.024354  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:14.024462  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.025321  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.136771  8576 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:13.968976 12796 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:13.993544 12979 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:03:14.026604  8732 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:14.017222  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.137872  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:14.027091 12879 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:14.027895 13016 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 13:03:14.027820  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:14.140614  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 318669us
I20260709 13:03:13.922818 12888 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.046172  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.047868  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:14.152678  8580 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:14.152649  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104855us
I20260709 13:03:14.154299  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.154922  8582 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:14.167444  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:14.065164 13024 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:14.168809  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.171516  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:14.172777  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.041035  8727 raft_consensus.cc:2889] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:14.040540  8658 rpcz_store.cc:276] 0709 13:03:12.438279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.839310 (+401031us) service_pool.cc:224] Handling call
0709 13:03:12.839898 (+   588us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:12.840224 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:12.840266 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:12.840933 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 13:03:12.964246 (+123313us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:12.964299 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.964412 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:13.973135 (+1008723us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:14.039834 (+ 66699us) raft_consensus.cc:1685] finished
0709 13:03:14.039856 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.040050 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4694912,"thread_start_us":122821,"threads_started":1,"wal-append.queue_time_us":263537}
I20260709 13:03:14.051649  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:14.054720  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.125540  8581 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:14.128362  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:14.131114  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 82048.9us
I20260709 13:03:14.139096  8584 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:14.138284  8577 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:14.175899  9330 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:14.028520  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.149981  8724 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:14.071120  8653 rpcz_store.cc:276] 0709 13:03:12.425247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.819403 (+394156us) service_pool.cc:224] Handling call
0709 13:03:12.819825 (+   422us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:12.820359 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:12.853456 (+ 33097us) log.cc:844] Serialized 14 byte log entry
0709 13:03:13.160086 (+306630us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:13.160624 (+   538us) log.cc:844] Serialized 52 byte log entry
0709 13:03:13.160769 (+   145us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:13.160814 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.160890 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:13.977498 (+816608us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:14.070218 (+ 92720us) raft_consensus.cc:1685] finished
0709 13:03:14.070365 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.070545 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65671,"prepare.queue_time_us":35361,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":91302,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":152,"spinlock_wait_cycles":8272640,"thread_start_us":221160,"threads_started":1,"wal-append.queue_time_us":259941}
I20260709 13:03:14.003188 12650 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:14.173822  8576 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.140705  8583 raft_consensus.cc:2376] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10. Re-sending same reply.
I20260709 13:03:14.175689  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:14.126850 12899 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:14.181342  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.181736  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:14.182116  8578 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.177819  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.183427  8651 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:14.184553  8657 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:14.179203  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:14.186993  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:14.187516  8698 leader_election.cc:400] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:14.187826  8698 leader_election.cc:403] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:14.188791  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:14.190253  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:14.193145  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.196702  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:14.204305  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:14.174613 12918 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.216777  8651 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:14.178498 13018 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:14.225925  8697 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:14.227505  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:14.140156 12455 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 13:03:14.221510  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:14.151692 12917 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 13:03:14.180465 12547 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:14.235206  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:14.233210  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:14.194504 12577 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:14.188668 13024 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:14.241379  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:14.255792  9322 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:14.256282  9330 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:14.249704  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:14.256799  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:14.257249  8727 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:14.249902  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.242185  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:14.249163 12997 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.658s	user 0.002s	sys 0.002s
I20260709 13:03:14.249127 12948 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:14.256382  9331 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:14.247491  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:14.260001  9331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:14.219470 12928 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:14.264506  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.277241  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:14.277776  8576 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:14.214640 12913 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:03:14.272085  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 162013us
I20260709 13:03:14.260691  9331 raft_consensus.cc:606] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
W20260709 13:03:14.275231  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 133729us
I20260709 13:03:14.275528  9330 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:14.275882  8655 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:14.224684 12968 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:14.218899 12963 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:14.225138 12853 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.260349 12917 consensus_queue.cc:1243] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:14.318789 12952 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:14.319141 13029 ts_tablet_manager.cc:933] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:14.296417  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 109472us
I20260709 13:03:14.234323 12974 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.194056 12951 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:03:14.300592  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:14.241883 12514 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:14.317272  8727 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:14.189216 12791 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 13:03:14.209661 12709 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
W20260709 13:03:14.309666  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 74994.4us
W20260709 13:03:14.295748  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 88186.9us
I20260709 13:03:14.346495  8576 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:03:14.346702  8727 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:14.310186  8577 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:14.211275 12879 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 13:03:14.312436  9330 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:14.260685 13024 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:14.364214  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 129101us
I20260709 13:03:14.322095  9322 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:14.365702  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:14.366407  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:14.338037 13013 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:03:14.275655 12354 raft_consensus.cc:2764] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:03:14.298382 12665 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:14.320715 12896 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.378274  8576 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:14.379350  8729 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:14.238161 12268 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.224506 12918 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:14.297093  8578 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index -1 to 7, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:03:14.194187 12531 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 13:03:14.370837  9327 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:14.337682  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:14.302052 12577 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:14.369675 13024 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:14.311137 12928 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:14.338092 13025 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:14.376354 12913 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:14.370090 12486 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.363766 12968 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:14.400148  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 62557.2us
I20260709 13:03:14.336712 12853 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:14.446211  9323 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:14.348721  8579 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:14.455224  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66680.9us
I20260709 13:03:14.387393 12796 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:14.462957  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 162476us
I20260709 13:03:14.377604 12665 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:14.542838  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:14.358739 12974 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:14.455575  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:14.491729 12994 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.553928  8576 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.474112  9322 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:14.554155  9323 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:14.479346 13043 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:14.480921 12952 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:14.557603  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.483203 12898 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.518926 12951 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:14.447890 13025 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:14.389531 12941 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:14.553342 13015 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:03:14.523314  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.524477  9326 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:14.574507  8577 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:14.466614 12926 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.537s	user 0.004s	sys 0.011s
W20260709 13:03:14.527961  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 233756us
I20260709 13:03:14.382422 12982 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.561456 13050 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 13:03:14.583519 12982 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:14.584434 12982 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:14.585064 13045 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:03:14.585935 13033 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260709 13:03:14.393878 12966 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.125s
I20260709 13:03:14.457890 12913 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:14.448905 12948 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:14.547693  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:14.549043 12665 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:14.551167 12635 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:03:14.552394 13030 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 13:03:14.388592 12896 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:14.477520 12577 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.470424 12486 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.482364 12917 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:14.558434  8581 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.390256 12629 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.568012  9328 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:14.567502  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:14.610765  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.611728  8581 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:14.395720 13027 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:14.535308  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.640625  8578 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:14.587852  9324 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:14.592000  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 123529us
I20260709 13:03:14.592813  8584 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:14.587064 12888 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:14.597095  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.574478 13025 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:14.613088  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.576380 12951 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:14.594280  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.654819  8581 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 6: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.594242 12913 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:14.568161 12994 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.652729  8577 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:14.645865  8578 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 13:03:14.650666  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:14.594312 12635 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:14.567354 12928 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.578950 12941 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:14.590579 12926 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.597158 12968 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:14.661031  8580 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:14.660274  8582 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:15.010030  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.010802  8578 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:14.772213  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 213342us
I20260709 13:03:14.707376  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2244) took 1187 ms. Trace:
W20260709 13:03:14.975034  8575 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.008s	sys 0.016s
W20260709 13:03:15.012521  8575 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.009s	sys 0.016s
I20260709 13:03:14.699868  8576 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:15.014434  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:14.681265 13042 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 13:03:14.662158  8577 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:03:14.913090  8584 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:14.661453  9322 raft_consensus.cc:606] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:14.724015  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:15.024834  9322 raft_consensus.cc:614] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:15.012084  8725 rpcz_store.cc:276] 0709 13:03:13.520178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.980044 (+459866us) service_pool.cc:224] Handling call
0709 13:03:13.980378 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:13.988946 (+  8568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.988977 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:13.989507 (+   530us) log.cc:844] Serialized 26 byte log entry
0709 13:03:14.152698 (+163191us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:14.152740 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.152817 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:14.707100 (+554283us) raft_consensus.cc:1685] finished
0709 13:03:14.707122 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.707302 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12568,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":162774,"threads_started":1,"wal-append.queue_time_us":331188}
I20260709 13:03:14.640132 13027 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:15.027256  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.028230  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.031507  8580 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:15.029047  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:14.604472 12547 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:14.961921 13031 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.397s	user 0.000s	sys 0.001s
I20260709 13:03:15.035679  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:14.685917 13050 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:15.050546  8650 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:15.047804  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:14.601508 13029 tablet_copy_client.cc:323] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:03:15.047993  8577 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 13:03:14.714397 12853 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:15.055866  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:14.681743 13034 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260709 13:03:15.003831 12629 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:15.009665 12918 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:14.726001 12973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:42016 (request call id 1) took 6709 ms. Trace:
I20260709 13:03:15.062278  8578 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:14.682935 12486 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:15.054138  8577 pending_rounds.cc:85] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 5
I20260709 13:03:14.653321 13043 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.393s
I20260709 13:03:15.076813  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 688) took 1112 ms. Trace:
W20260709 13:03:15.075277 13051 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.348s	user 0.001s	sys 0.000s
I20260709 13:03:14.984809 12951 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.080271  8659 rpcz_store.cc:276] 0709 13:03:13.964331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.187747 (+223416us) service_pool.cc:224] Handling call
0709 13:03:14.209680 (+ 21933us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:14.210010 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:14.210045 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:14.210545 (+   500us) log.cc:844] Serialized 26 byte log entry
0709 13:03:14.343307 (+132762us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:14.343334 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.343435 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.076569 (+733134us) raft_consensus.cc:1685] finished
0709 13:03:15.076597 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.076753 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4488,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":22356,"thread_start_us":132483,"threads_started":1,"wal-append.queue_time_us":341960}
I20260709 13:03:15.031062 13055 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:15.084390  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 433005us
I20260709 13:03:15.042508 12898 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:15.088245  8582 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:15.091032  8577 pending_rounds.cc:107] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:03:15.091364  8577 raft_consensus.cc:2889] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:14.706686 12901 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.352s
I20260709 13:03:15.091840  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1009) took 1148 ms. Trace:
I20260709 13:03:15.050022 13050 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:15.096520  9294 leader_election.cc:400] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9.
I20260709 13:03:15.100417  9294 leader_election.cc:403] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 13:03:15.100915  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:15.101608  8578 raft_consensus.cc:2376] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10. Re-sending same reply.
I20260709 13:03:15.084102 12974 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:15.083344 13055 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.100229  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:15.094854  9325 rpcz_store.cc:276] 0709 13:03:13.943520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.118849 (+175329us) service_pool.cc:224] Handling call
0709 13:03:14.119243 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:14.119528 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:14.299420 (+179892us) log.cc:844] Serialized 14 byte log entry
0709 13:03:15.091413 (+791993us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:15.091477 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:15.091548 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.091637 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.091769 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54269,"prepare.queue_time_us":225255,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":119922,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":68022,"thread_start_us":916764,"threads_started":3,"wal-append.queue_time_us":355807}
I20260709 13:03:15.108472  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:15.112685  8655 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:15.094975 12899 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:15.115016 12899 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:15.115937 12899 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.114953 12650 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:15.119374  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:15.099767 12629 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:03:15.101188 13041 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:03:15.127556  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1010) took 1183 ms. Trace:
I20260709 13:03:15.127367 12547 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.128051  9329 rpcz_store.cc:276] 0709 13:03:13.943808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.124824 (+181016us) service_pool.cc:224] Handling call
0709 13:03:14.168960 (+ 44136us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:14.169311 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:14.317654 (+148343us) spinlock_profiling.cc:243] Waited 8.36 ms on lock 0x7b1800083800. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:03:14.559472 (+241818us) log.cc:844] Serialized 14 byte log entry
0709 13:03:14.833654 (+274182us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.015221 (+181567us) log.cc:844] Serialized 26 byte log entry
0709 13:03:15.015374 (+   153us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:15.015410 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.015478 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.127205 (+111727us) raft_consensus.cc:1685] finished
0709 13:03:15.127333 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.127491 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":226307,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":251841,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":8205,"spinlock_wait_cycles":19228928,"thread_start_us":494508,"threads_started":2,"wal-append.queue_time_us":482704}
I20260709 13:03:15.131145 12901 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:15.131613 12901 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:15.132438 12901 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.135143  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:15.135566  8659 raft_consensus.cc:2410] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:03:15.138124  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.141564  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:15.142480  8581 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:15.137652 12889 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:15.142987 13050 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.143703 12889 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:15.143895  8575 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:15.095335 12896 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:15.144016 13050 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:15.145287 13024 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.000s
I20260709 13:03:15.154636  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:15.170507  8575 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:03:15.158257 12941 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.161190 12926 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:15.162467  8582 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:15.182111  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:15.162801  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.183501  8576 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:15.163362  8580 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 13:03:15.164222  8584 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:15.162467  8654 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:15.150343  8578 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:15.185479  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.133984 12898 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9.
I20260709 13:03:15.183053  8582 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:15.186803  8578 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:15.173430  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.185284  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2709) took 1238 ms. Trace:
I20260709 13:03:15.188946  8580 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:15.190886  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:15.198527  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.207748  8582 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:15.185314  8549 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.692s (SENT)
I20260709 13:03:15.104214 12635 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.079530 12973 rpcz_store.cc:276] 0709 13:03:07.950101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.998059 (+ 47958us) service_pool.cc:224] Handling call
0709 13:03:10.235466 (+2237407us) log.cc:844] Serialized 492 byte log entry
0709 13:03:10.353078 (+117612us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54000898d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1c735 00007f140ebdcd29 00007f140815b1d4 00007f140815b162 00007f140815b091 00007f140815b00c 00007f140815a2c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:14.655666 (+4302588us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2932,"raft.run_cpu_time_us":1721,"raft.run_wall_time_us":1720,"spinlock_wait_cycles":1132225280,"thread_start_us":51287,"threads_started":1,"wal-append.queue_time_us":100059,"wal-append.run_cpu_time_us":1999,"wal-append.run_wall_time_us":1004972}
I20260709 13:03:15.210093 12635 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:15.210548  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:15.196683 12895 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:15.185840 13024 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.205103  8654 rpcz_store.cc:276] 0709 13:03:13.946761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.078746 (+131985us) service_pool.cc:224] Handling call
0709 13:03:15.161951 (+1083205us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:15.162140 (+   189us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:15.185059 (+ 22919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.185235 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":112364928}
I20260709 13:03:15.217780  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:15.218297  8725 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:15.224187  8580 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:15.221765  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:15.183008 12901 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:15.228669  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55377us
I20260709 13:03:15.232548  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:15.233338  8578 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:03:15.238824  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:15.217383 13025 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.550s	user 0.006s	sys 0.000s
I20260709 13:03:15.250289 13060 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:15.251526  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:15.252398  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 67263.5us
I20260709 13:03:15.254081 13024 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.221901 12895 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:15.258402  8575 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:15.231339 12898 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.262115  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.239759 13041 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:15.234185 13065 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:15.260994 12895 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:15.319597  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 817) took 1889 ms. Trace:
I20260709 13:03:15.320039  8728 rpcz_store.cc:276] 0709 13:03:13.430363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.819279 (+388916us) service_pool.cc:224] Handling call
0709 13:03:13.825936 (+  6657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:14.190110 (+364174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:14.190155 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:14.190211 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:14.190278 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.319362 (+1129084us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b5400417154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:15.319414 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.319544 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":836899200}
I20260709 13:03:15.322420  8656 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:15.323254  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2708) took 1376 ms. Trace:
I20260709 13:03:15.323652  8656 rpcz_store.cc:276] 0709 13:03:13.946424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.071103 (+124679us) service_pool.cc:224] Handling call
0709 13:03:15.322006 (+1250903us) spinlock_profiling.cc:243] Waited 56.6 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:15.322205 (+   199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:15.322965 (+   760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.323188 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":130255232}
I20260709 13:03:15.275962 13060 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:15.260635 13025 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:15.329303  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1003) took 1385 ms (client timeout 1511 ms). Trace:
I20260709 13:03:15.294864 12951 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:15.329993  9324 rpcz_store.cc:269] 0709 13:03:13.941739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.976337 (+ 34598us) service_pool.cc:224] Handling call
0709 13:03:15.326414 (+1350077us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b5400551c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:15.326614 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1242224512}
I20260709 13:03:15.268571 12898 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:15.343169  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 170285us
W20260709 13:03:15.351540 12665 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.750s	user 0.009s	sys 0.000s
I20260709 13:03:15.356662 12898 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.357414  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2675) took 1095 ms. Trace:
I20260709 13:03:15.358349 13055 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:15.364151  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:15.368248  8583 rpcz_store.cc:276] 0709 13:03:14.261559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.284111 (+ 22552us) service_pool.cc:224] Handling call
0709 13:03:14.336508 (+ 52397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:14.373905 (+ 37397us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:14.930604 (+556699us) log.cc:844] Serialized 14 byte log entry
0709 13:03:15.227738 (+297134us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.240389 (+ 12651us) log.cc:844] Serialized 26 byte log entry
0709 13:03:15.240534 (+   145us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:15.241310 (+   776us) log.cc:844] Serialized 14 byte log entry
0709 13:03:15.241630 (+   320us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.241734 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.357131 (+115397us) raft_consensus.cc:1685] finished
0709 13:03:15.357160 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.357330 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26717,"prepare.queue_time_us":196244,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":1934,"raft.queue_time_us":48103,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":8483,"thread_start_us":683967,"threads_started":2,"wal-append.queue_time_us":276695}
I20260709 13:03:15.374195 12547 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:15.378278  8657 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:15.378887  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:15.379566  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:15.381067  8728 pending_rounds.cc:85] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 5
I20260709 13:03:15.382153  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:15.382444  8729 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:03:15.392499  8728 pending_rounds.cc:107] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 13:03:15.383116  8654 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:15.392803  8728 raft_consensus.cc:2889] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:15.371325 12796 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.387101  8726 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 13:03:15.395395  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 133391us
W20260709 13:03:15.361145 12879 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.964s	user 0.018s	sys 0.026s
I20260709 13:03:15.394922  8655 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:15.395936  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:15.361747 12889 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.396791  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:15.414235  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:15.402045  8658 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:15.396958  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:15.469748  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 72880.8us
I20260709 13:03:15.470463  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:15.470593  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:15.414580  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.472041  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.472201  8654 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:03:15.476747  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.475618  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:15.440091 12879 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.023s	sys 0.027s
I20260709 13:03:15.488672  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:15.470438  8729 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:15.486210  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.403365 12926 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.402238 12898 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:15.496191 12898 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:15.468034 12547 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.498843  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.402000 12268 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.391525 12665 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.483023 13055 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:15.478574 13060 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:15.485926 12968 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:15.467046 12928 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:03:15.417217 12796 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.475077 12889 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:15.498522 13061 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:15.544689  8730 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:15.501118 12963 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:15.507428  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 255937us
I20260709 13:03:15.559990  8729 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:03:15.562083  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:15.576737 13068 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 13:03:15.577907  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39404 (request call id 9) took 1104 ms. Trace:
I20260709 13:03:15.578411  9322 rpcz_store.cc:276] 0709 13:03:14.473807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.473968 (+   161us) service_pool.cc:224] Handling call
0709 13:03:15.577687 (+1103719us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54006c2d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c10853 00007f140ebde60a 00007f14081647d4 00007f1408164762 00007f1408164691 00007f140816460c 00007f14081638c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:15.577819 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":245048192}
I20260709 13:03:15.515053 12898 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.583811  8726 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:15.541960 12577 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:15.516060 12629 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
W20260709 13:03:15.595214 13070 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:03:15.595369 13027 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:15.598071 13032 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.140s	user 0.014s	sys 0.005s
I20260709 13:03:15.598623 12665 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:15.599192 13061 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.601498 13076 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.605660 13025 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.605989  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2511) took 1699 ms. Trace:
I20260709 13:03:15.624853  8650 rpcz_store.cc:276] 0709 13:03:13.906743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.909612 (+  2869us) service_pool.cc:224] Handling call
0709 13:03:13.920545 (+ 10933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:15.055150 (+1134605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.605679 (+550529us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540018fbd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:15.605910 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2597997440}
I20260709 13:03:15.606204 12889 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:15.607600  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2653) took 1601 ms. Trace:
W20260709 13:03:15.581971 13071 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:03:15.611603  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.630802  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.502789 12926 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.507121 12547 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.546060 13055 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.066s
I20260709 13:03:15.597159 12796 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:15.626915  8579 rpcz_store.cc:276] 0709 13:03:14.006453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.006605 (+   152us) service_pool.cc:224] Handling call
0709 13:03:15.607330 (+1600725us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54000b6154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:15.607513 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":312011520}
I20260709 13:03:15.551050 12963 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:15.661316 12171 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:15.636495  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:15.622694 13075 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:15.639355 13012 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:15.640817 12895 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:15.643033  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.670002  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:15.670818  8725 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
W20260709 13:03:15.643903  8622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.870s (SENT) [suppressed 19 similar messages]
I20260709 13:03:15.672297  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:15.673957  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:15.644621  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:03:15.645200  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:15.647733  9328 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 780) took 1464 ms (client timeout 1662 ms). Trace:
W20260709 13:03:15.679306  9328 rpcz_store.cc:269] 0709 13:03:14.183745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.187124 (+  3379us) service_pool.cc:224] Handling call
0709 13:03:15.647571 (+1460447us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b5400816354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:15.647678 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779776768}
I20260709 13:03:15.648329 12455 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:15.648938  8742 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet f4f350f118fc449fb61357528a6b14f7 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:03:15.623907 13041 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:15.514215 12268 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.663419  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:15.668045  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 165600us
I20260709 13:03:15.658577 12889 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:15.683918  9322 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:03:15.693781  9323 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:15.662904 13027 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:15.703989  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.666765 13012 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:15.707275  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.717303  8696 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.718678  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.718897  8577 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:15.718851  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.719780  8581 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:15.719990  8584 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:15.663466 12577 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 13:03:15.628324 13061 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:15.704941  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:15.725564  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.726074  8579 raft_consensus.cc:2376] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9. Re-sending same reply.
I20260709 13:03:15.727142  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.727735  8725 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:15.727881  8577 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:15.710003  9328 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:15.631556 13076 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.691524 12455 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:15.700196 12896 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:15.723831  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.732406  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.680495 12635 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:15.668852 12665 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:15.728756  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:15.702553 12913 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.026s	user 0.007s	sys 0.006s
I20260709 13:03:15.713968  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:15.701665 13050 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:15.708015  9326 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:15.713771 12973 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:15.737231  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.745155  8579 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:15.743317  9327 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:15.714337 12889 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:15.702252  8742 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet f4f350f118fc449fb61357528a6b14f7 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-f4f350f118fc449fb61357528a6b14f7
I20260709 13:03:15.751475  8578 raft_consensus.cc:2410] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:03:15.728101 12577 consensus_queue.cc:1243] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:03:15.751950  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:15.755735  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 164702us
I20260709 13:03:15.750007 12973 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:15.722676 12941 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 13:03:15.732486 12888 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:15.746735 12635 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.752924 12889 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:15.831948 12973 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:15.746872 12913 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.750603 13050 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.792306 13073 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:15.844192 12888 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:15.875921 12888 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:15.879519 13045 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:15.880141 13045 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:15.882772 13045 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.883586  8742 tablet_copy_source_session.cc:215] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:03:15.886193 12895 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:15.849139 13088 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:15.892222 13088 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:15.893114 12635 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.841843 13012 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:15.872294 12665 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:15.895527 13088 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.918035  8732 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:03:15.906219  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:03:15.925920  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 243904us
I20260709 13:03:15.927265  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:15.928923  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:15.930523  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:15.931202  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 213993us
I20260709 13:03:15.932037  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.933027  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:15.933194  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:15.934181  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 279527us
W20260709 13:03:15.918723 13079 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20260709 13:03:15.948560  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:15.829540 13065 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.897346 13012 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:15.914637 12455 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:16.027350  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:16.049906  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 101465us
W20260709 13:03:16.051433  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.053542  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:16.055125  8578 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:16.061966  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:16.067953  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:16.068885  8582 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:16.071957  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:16.072197  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.916265 12913 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:15.908119 12547 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.077984  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:16.079035  8578 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:15.902681 13076 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:16.079422  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:15.903738 13061 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.914925 12926 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:16.081377  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:16.082338  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.082808  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:16.156723  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.947405 12532 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:15.947880 12928 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:15.952680 12895 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:16.166740 13089 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 13:03:16.167485 12889 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:16.168483 13084 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
W20260709 13:03:16.170174 13085 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260709 13:03:16.173110 12895 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:16.174006 12889 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.176473 13076 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:16.177219 13076 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:16.178365 12888 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.180399 13076 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.180961 13096 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:16.181118 13097 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:03:16.182967 13094 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 13:03:16.183215 13097 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:16.186585 12888 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:16.085336 12455 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:16.138100 13095 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:16.085736  8580 raft_consensus.cc:2376] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10. Re-sending same reply.
I20260709 13:03:15.935173 12268 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:16.158404  8579 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:16.087832  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:15.979267 13087 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:15.971484 13050 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:15.927147 13073 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:16.232626  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 297541us
I20260709 13:03:16.220144  8577 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:16.078380 12577 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.907496 12665 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:16.230756  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 297755us
I20260709 13:03:16.078924 12963 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.081768 13099 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:16.219468 12913 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:03:16.240453 13027 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.525s	user 0.010s	sys 0.003s
I20260709 13:03:16.224342 13107 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:16.083653 13065 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.259183 12577 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:16.269547  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 213813us
I20260709 13:03:16.272122  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:16.273540 12913 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:16.273958 12913 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:16.274348 12913 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:16.275388 13054 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:16.275852 13054 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:16.267407 13107 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.286298  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:16.234728 13087 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.287649 12635 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:16.308538  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 50218.2us
I20260709 13:03:16.305091  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2549) took 1075 ms. Trace:
I20260709 13:03:16.310079  8730 rpcz_store.cc:276] 0709 13:03:15.229265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.229584 (+   319us) service_pool.cc:224] Handling call
0709 13:03:15.248638 (+ 19054us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:15.545240 (+296602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.304732 (+759492us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54000e9f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:16.305003 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":680098560}
W20260709 13:03:16.312148  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 53929.8us
I20260709 13:03:16.286048 13107 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:16.316234  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:16.296168 12635 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.319041 13087 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.319065 12635 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.305682 13102 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:16.329447  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.276813 13054 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.334098 12577 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.277519 13105 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 13:03:16.282657 12963 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 13:03:16.285245 12975 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
W20260709 13:03:16.344820 13048 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.969s	user 0.001s	sys 0.003s
I20260709 13:03:16.347628  8584 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:03:16.347775 12665 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.349097 13065 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.339054 12497 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.350231 12497 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.352264 12497 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.352491 12973 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.337435 13102 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.330552 13108 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:16.358878  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:16.364586  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:16.342027 13054 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.361568 12895 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:16.371732  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:16.359746 13108 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:16.363022 12928 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:16.369196 12895 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:16.380237  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:16.380964  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:16.365944 12268 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:16.381942  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:16.381901  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:16.383549  8577 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:16.384022  8584 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:16.375351 13105 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.011s
I20260709 13:03:16.386260 13105 consensus_queue.cc:1243] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:03:16.367355 12963 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:16.387351  8580 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:16.388614  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 73807us
I20260709 13:03:16.374781 12888 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:16.377964 13095 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:03:16.393627  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 50657.4us
I20260709 13:03:16.379045 13111 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:16.382622 12975 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:16.393280  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:16.376922 13108 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:16.401809  8577 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:16.406803  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:16.390004 13114 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.387012 13073 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.393802 12969 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:16.382499 12665 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260709 13:03:16.420607 12895 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.422015 12895 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:16.392498 12268 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:16.393635 12888 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:16.398027 13111 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:16.410466 12975 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.395351 13109 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 13:03:16.397167 13106 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:03:16.447374  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:16.448079  8577 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:16.428233 13114 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.452800 13114 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:16.429126 13095 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:16.445017  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:16.461416  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:16.416990 13096 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 13:03:16.434182 12888 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:16.417263 12928 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 13:03:16.443147 12975 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:16.473868  8577 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
W20260709 13:03:16.474675  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.465863 13095 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.469141 13114 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:16.473660 13115 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:16.478586 12973 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.534583 12973 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:16.535213 12973 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:16.536303 12973 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.476110 13106 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
W20260709 13:03:16.538082  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.540025 12896 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:16.540681 12896 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:16.507812 13114 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.533686 12898 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.504421 13095 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:16.553342  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 139569us
I20260709 13:03:16.510396 13115 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:16.555137  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 137098us
I20260709 13:03:16.545774 13114 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:16.559475  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:16.559412 12963 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:16.503278 13111 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.558759 13115 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:16.668630  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 109081us
I20260709 13:03:16.670413  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:16.669691  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:16.551573 12898 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.638209 12577 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.544490 12268 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.688180 12994 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:16.689671  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 130337us
I20260709 13:03:16.692234  8726 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:16.693037  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.560104 12941 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.114s
I20260709 13:03:16.693670  8727 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:16.693717  8731 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:16.694550  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.673322 12532 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:16.563364 12913 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.710284  8730 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20260709 13:03:16.712931  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:16.699167 13100 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.400s	user 0.001s	sys 0.003s
I20260709 13:03:16.676040 13045 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.024s
W20260709 13:03:16.699124 13117 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 13:03:16.719261 12928 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:16.720268 12898 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.723546  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2260) took 1343 ms. Trace:
I20260709 13:03:16.725476 13122 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 13:03:16.726402 13121 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:03:16.708905 13113 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 13:03:16.727208 12963 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.696219 13063 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:16.752526 13063 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:16.753087  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.753131 13063 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.755154  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:03:16.728632  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet b063565a0d8244cabf980570d2fc8a38: Illegal state: Replica a0d703ee18464475a290425976246555 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:03:16.729046 13107 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.730489 13029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f350f118fc449fb61357528a6b14f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:03:16.695266 12577 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.758471 13107 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.759294 13107 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:16.759739 13107 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.760185 13107 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:16.706195 12532 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:16.745357 13124 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:16.762066 13124 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.762306 12532 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.764008 12532 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:16.732487 12969 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:16.698216 13111 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:16.707698 13099 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:16.740633  8732 rpcz_store.cc:276] 0709 13:03:15.380551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.382821 (+  2270us) service_pool.cc:224] Handling call
0709 13:03:15.383111 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:15.926750 (+543639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.723252 (+796502us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b5400472cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:16.723478 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1229494528}
I20260709 13:03:16.737216 12928 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.773360  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1037) took 1103 ms. Trace:
I20260709 13:03:16.802944  9324 rpcz_store.cc:276] 0709 13:03:15.670208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.674958 (+  4750us) service_pool.cc:224] Handling call
0709 13:03:15.704039 (+ 29081us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:15.715686 (+ 11647us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:15.715717 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.727813 (+ 12096us) log.cc:844] Serialized 26 byte log entry
0709 13:03:16.079182 (+351369us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:16.079225 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.079299 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:16.452056 (+372757us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b48001b06d8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:16.687669 (+235613us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:16.772996 (+ 85327us) raft_consensus.cc:1685] finished
0709 13:03:16.773028 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.773271 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6300,"raft.queue_time_us":9181,"raft.run_cpu_time_us":2860,"raft.run_wall_time_us":736022,"spinlock_wait_cycles":20195200,"thread_start_us":322113,"threads_started":1,"wal-append.queue_time_us":512991}
I20260709 13:03:16.759945 12577 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.769147 12969 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:16.769328 12898 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.788748 13095 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.795404 12941 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:16.801959  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2262) took 1420 ms. Trace:
I20260709 13:03:16.746212 13120 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.832231 12896 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.844432 12547 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.845185 13043 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.845773 13043 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.851891  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:16.857939  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.860288  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.861136  8656 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:16.805900 12928 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:16.797252 12913 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.862936  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.873078  8656 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
W20260709 13:03:16.875257  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 204910us
I20260709 13:03:16.867059  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.867375  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.867707  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.869925  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.878710  8658 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:16.868276  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:16.863858  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 191917us
I20260709 13:03:16.878726  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:16.863420 13124 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.898165 13043 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.900319  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:16.900501 13095 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.863446 12901 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:16.901723 12901 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.903038 13095 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:16.903775 12901 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.904173 12888 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.905076  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.905917  8659 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:16.906652  8653 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:16.881882  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:16.880775  8654 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:16.887493  8728 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 13:03:16.888092  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:16.886344  8732 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:16.911315  8652 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 7.
W20260709 13:03:16.888536  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.848114 13120 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.872332 12577 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.893523 13108 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.890650 13114 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.863190 13045 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:16.871699 12665 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:16.889853 12975 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.865272 13073 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 13:03:16.919813 13107 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.920831 13095 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:16.909865 13075 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:16.829869 12898 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.825913  8733 rpcz_store.cc:276] 0709 13:03:15.381132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.384557 (+  3425us) service_pool.cc:224] Handling call
0709 13:03:15.402530 (+ 17973us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:15.468509 (+ 65979us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:15.468539 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.477490 (+  8951us) log.cc:844] Serialized 26 byte log entry
0709 13:03:16.338594 (+861104us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b1800405fa0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:16.391849 (+ 53255us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:16.391919 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.392045 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:16.801662 (+409617us) raft_consensus.cc:1685] finished
0709 13:03:16.801682 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.801893 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":73274,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":5659,"spinlock_wait_cycles":16555264,"thread_start_us":53014,"threads_started":1,"wal-append.queue_time_us":1000289}
I20260709 13:03:16.863221 12547 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.923022 13111 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:03:16.928082 12889 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:16.934002  8726 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:16.935575  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.942492  8725 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:16.943781  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.870822  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:16.944355  8731 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:16.945356  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.943848  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:16.946768  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 76056.9us
I20260709 13:03:16.947341  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.947353  8729 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:16.925271 12532 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 13:03:16.955837  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:16.962544  8733 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
W20260709 13:03:16.947568  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 192492us
I20260709 13:03:16.964053  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:16.964774 13111 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:16.965229 13111 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.965312 12547 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:16.965705 13111 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.944701 13075 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:16.967500  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:16.979730  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.943917 12889 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:16.986342 13124 raft_consensus.cc:2764] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:03:16.986784 12889 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.988554 12889 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:16.966300 12941 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:16.989789 12941 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:16.990600 12941 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:16.991777 12941 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:16.992213 12975 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:16.993044 13131 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:16.994635 13131 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:16.994989 13131 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:16.995833 13131 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.945297 12665 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:16.998829  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:16.999661  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet 06f98a166dd945b39554b2f278c933ef: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:03:16.933384 12577 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.958894 13095 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:16.997179 13131 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:17.012090  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:16.943495 13063 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:16.944988 13045 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:17.028774 13107 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:17.029373 13045 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:17.030048 13045 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.029731 13107 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:17.030573 13045 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.031957 12665 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:17.032845 13045 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.034225 13045 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:17.034907 12577 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.035008 13045 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:17.035506 12577 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.036036 12577 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:17.037824 13054 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.667s	user 0.006s	sys 0.006s
I20260709 13:03:17.039346 12913 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:17.039968 12913 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:17.044064 13063 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:17.045926 12898 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:17.011574 12975 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:16.990624 12888 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:17.041097 12913 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:17.019789 12969 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:17.024838  8733 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:17.055572  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:17.057034  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.999328 13136 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:17.080734 13102 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.089465 12994 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:17.090137 12994 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:17.091295 12994 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.092875 12994 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:17.095386  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.095572 12577 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:17.101967 13049 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:17.102592 13049 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:17.103677 13114 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:17.107576  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.410s	user 0.024s	sys 0.059s
I20260709 13:03:17.103704 13049 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:17.107926  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.411s	user 0.024s	sys 0.059s
I20260709 13:03:17.108220 13049 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:17.015516 13124 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.095s
W20260709 13:03:17.108258  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 1.41095e+06us
I20260709 13:03:17.121284  8728 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:17.123178  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 259599us
I20260709 13:03:17.125065  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:17.141144  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:17.169199  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.170465  9324 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:17.170965  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:17.062683 12898 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.181031  9330 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 13:03:17.175697  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.183634  9323 raft_consensus.cc:2410] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 13:03:17.183923  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.184767  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.185051  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.177592  9325 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:17.185539 12171 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.179657  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.186053 12171 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.174408  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.185073  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:17.187518  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.188539  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.189103  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.189906  9331 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:17.192284  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.058663 12969 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:17.198756  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.199715  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.200536  9325 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:17.201093  8733 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:17.202105  8658 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:17.204595  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.205231  9323 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:17.198289  9331 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:17.192302  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.209827  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.210173  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:17.210367  9329 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:17.210786  8728 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:17.212898  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.210969  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.214548  9325 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:17.214718  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.127698 13029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:43664 (request call id 765) took 3148 ms. Trace:
I20260709 13:03:17.216529  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.217243  9324 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:17.218696  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.222397 12171 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.190008 13059 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:17.197712 13116 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:17.190305 13054 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:17.193483 13073 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:17.240955  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.228652  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.243659  8724 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:03:17.193928 13105 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:17.194701 12547 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:17.254839 12171 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:17.235074  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.457262  9330 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:03:17.458216  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.458926  9330 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
W20260709 13:03:17.222610 13119 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.013s	sys 0.013s
I20260709 13:03:17.261090  8730 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:17.262319  9322 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:17.263772  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 307969us
I20260709 13:03:17.461967  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:17.344235  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.198369 12898 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.464123 12171 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:17.465700  9327 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:03:17.432477  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 449394us
W20260709 13:03:17.467947  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:17.444300  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.167806 13063 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:03:17.450443  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:17.451900  9324 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:17.452888  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.485203  9329 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:17.274745  8729 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:03:17.454723  8728 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:17.486958  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.487916  8728 raft_consensus.cc:2410] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
W20260709 13:03:17.455350  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 555022us
I20260709 13:03:17.489317  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:17.490447  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:17.491299 13111 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.525s	user 0.014s	sys 0.000s
I20260709 13:03:17.491516  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:17.491770 13111 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:17.454226  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.487808  8656 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:17.496295  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:17.478801 12268 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.002s	sys 0.028s
I20260709 13:03:17.514671  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:17.460672 13119 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.014s	sys 0.013s
I20260709 13:03:17.518273  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:17.519268  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.018s	sys 0.026s
W20260709 13:03:17.525182  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.018s	sys 0.026s
W20260709 13:03:17.525863  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 507414us
I20260709 13:03:17.530673  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:17.466918 13102 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.467244 13137 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.482273 12577 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.342267 13073 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.478931 12898 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:17.548481  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.450809 13097 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.127s
W20260709 13:03:17.520723 12268 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.003s	sys 0.028s
W20260709 13:03:17.521382 12928 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.645s	user 0.009s	sys 0.000s
I20260709 13:03:17.620142 13147 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:03:17.619966 12928 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:17.536841 13115 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.861s	user 0.007s	sys 0.012s
W20260709 13:03:17.621524 13139 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:03:17.621783 13115 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.623111  9323 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:17.624240 13115 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.625867  8655 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:17.467274 13105 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.520833 13108 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:17.568022  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.566426  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.362171 13133 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.201s
I20260709 13:03:17.479530 13116 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260709 13:03:17.342751 13129 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.302s
W20260709 13:03:17.580152 12926 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.278s	user 0.017s	sys 0.016s
I20260709 13:03:17.563777 13125 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:17.617985  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:17.619283 13144 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 13:03:17.619191 12268 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:17.578166 12969 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:03:17.604228 13132 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.464s	user 0.000s	sys 0.001s
I20260709 13:03:17.646272  8728 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:17.669057 13149 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:17.648100  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 389837us
W20260709 13:03:17.656999  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 202063us
I20260709 13:03:17.223085 13029 rpcz_store.cc:276] 0709 13:03:13.920708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.921805 (+  1097us) service_pool.cc:224] Handling call
0709 13:03:14.319516 (+397711us) ts_tablet_manager.cc:934] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
0709 13:03:16.679872 (+2360356us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:03:17.028131 (+348259us) tablet_metadata.cc:665] Metadata flushed
0709 13:03:17.028274 (+   143us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:03:17.068291 (+ 40017us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:03:17.068749 (+   458us) tablet_replica.cc:206] Creating consensus instance
0709 13:03:17.069229 (+   480us) ts_tablet_manager.cc:989] Replica registered.
0709 13:03:17.069325 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":27895,"tablet-copy.queue_time_us":297144,"thread_start_us":425848,"threads_started":2}
I20260709 13:03:17.709506 13029 tablet_copy_client.cc:806] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:03:17.710269 13029 tablet_copy_client.cc:670] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
W20260709 13:03:17.711796  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.598412 13102 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:17.560642 13122 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:03:17.678272  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 107856us
W20260709 13:03:17.687500  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 169313us
I20260709 13:03:17.688763  8729 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:17.636291 13073 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:17.658524 13143 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:17.603629 12577 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.699379  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.646986  8730 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:17.700606  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 250250us
W20260709 13:03:17.682056 12926 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.403s	user 0.019s	sys 0.016s
I20260709 13:03:17.672165 13111 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.597922 13146 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.862006  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2524) took 1007 ms. Trace:
I20260709 13:03:17.719944  9323 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:17.722406  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.660908 13147 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
W20260709 13:03:17.727030  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 268921us
W20260709 13:03:17.717733 13095 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.697s	user 0.017s	sys 0.000s
W20260709 13:03:17.784237 13075 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.771s	user 0.006s	sys 0.003s
I20260709 13:03:17.775830 13150 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:17.682219 13137 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.765501 13165 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:17.769925 13155 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:03:17.764755 13167 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:17.770366 13105 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.764304 13054 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.858125 13143 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:17.772964 13102 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:17.722330 13149 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:17.767336 12928 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:17.784237  9325 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.569s	user 0.006s	sys 0.003s
I20260709 13:03:17.766527 13115 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:17.773819  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.882861 13099 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.896471  8733 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
W20260709 13:03:17.798897 13153 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.140s	user 0.001s	sys 0.001s
I20260709 13:03:17.884466 13095 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:03:17.789189 13049 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.680s	user 0.009s	sys 0.003s
I20260709 13:03:17.765676  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2278) took 1062 ms. Trace:
I20260709 13:03:17.901386  8727 rpcz_store.cc:276] 0709 13:03:16.704029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.707295 (+  3266us) service_pool.cc:224] Handling call
0709 13:03:16.707486 (+   191us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:16.726526 (+ 19040us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.726558 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:16.727075 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 13:03:16.948094 (+221019us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:16.948135 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.948192 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:17.765453 (+817261us) raft_consensus.cc:1685] finished
0709 13:03:17.765474 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.765613 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18667,"raft.queue_time_us":19900,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":219715,"threads_started":1,"wal-append.queue_time_us":394002}
I20260709 13:03:17.761911 13148 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.829209 12901 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:17.834496 13126 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.108s	user 0.000s	sys 0.004s
W20260709 13:03:17.835901 12994 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.743s	user 0.011s	sys 0.002s
I20260709 13:03:17.839712  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.908008  8654 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
W20260709 13:03:17.840350 13131 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.819s	user 0.006s	sys 0.004s
I20260709 13:03:17.840481 12268 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:17.842823 12889 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.854s	user 0.008s	sys 0.006s
I20260709 13:03:17.843299  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2275) took 1149 ms. Trace:
I20260709 13:03:17.898623 13102 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.846364 12497 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:17.895089 13054 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.912536  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.874927 13146 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.848897 13096 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:17.828269 13116 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:17.920481  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.850576 12896 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:17.922736  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65221.9us
I20260709 13:03:17.856619  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:17.855687  9297 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:17.923480  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66918.7us
W20260709 13:03:17.923614  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67979.9us
I20260709 13:03:17.860252  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:17.855123 13122 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:17.871549 12577 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.873471 13136 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:17.873986  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.874876  8698 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:17.932842  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:17.875628  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.877295  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.879069  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.933728  8584 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:17.881194  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:17.882922  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.862703  8657 rpcz_store.cc:276] 0709 13:03:16.854185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.858270 (+  4085us) service_pool.cc:224] Handling call
0709 13:03:16.875188 (+ 16918us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:17.221704 (+346516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.221738 (+    34us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:17.351722 (+129984us) log.cc:844] Serialized 78 byte log entry
0709 13:03:17.702435 (+350713us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:03:17.861453 (+159018us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.861708 (+   255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.861777 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:17.861822 (+    45us) raft_consensus.cc:1685] finished
0709 13:03:17.861833 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.861954 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29246,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":15829,"thread_start_us":523510,"threads_started":2,"wal-append.queue_time_us":350077}
I20260709 13:03:17.885967  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.889801  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.936143  8579 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 13:03:17.936261  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 61478.9us
W20260709 13:03:17.937162  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.894687  9325 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:17.885701 13150 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:17.938592  8577 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:17.938845  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.914000 12497 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:17.892114 13143 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:17.114496  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:17.911762  8724 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:17.947268  8584 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:17.948032  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:17.912086  8653 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:17.847716 13156 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:17.887178 13075 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.910669  8726 rpcz_store.cc:276] 0709 13:03:16.693577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.694212 (+   635us) service_pool.cc:224] Handling call
0709 13:03:16.694495 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.942273 (+247778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.842955 (+900682us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54002049d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:17.843218 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":550205696}
I20260709 13:03:17.907891  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:17.918169  8725 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:17.849643  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 141431us
I20260709 13:03:17.925057  8729 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:17.903259 13099 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.962103  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.932307  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.910364 12901 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.938666  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.963835  9329 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:17.882910 13133 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 13:03:17.941656  8580 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:17.942955  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:17.968657  9325 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:17.918561 13102 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:17.945711  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.896224 13115 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:17.956075  8584 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: []
I20260709 13:03:17.922281 13116 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:17.943315 13150 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:17.930703  8733 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:17.879554  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.984812  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:17.985711  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.893744 13165 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.933146 13073 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:17.988370  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.949972 13156 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:17.899288  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:17.965162  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:17.993750  8577 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:17.910712 13125 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:17.971050  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:17.931197 13120 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.941146 12889 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:17.907848 13148 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.975050  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.998725  8650 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:17.998250 13125 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.962172  8727 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:17.999673  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.981833  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:17.998222 12889 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:18.000758  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:17.954432 12577 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.945938 12928 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:18.001731 12928 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:17.998006  8579 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:17.963184  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:18.001415  8650 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:17.990929  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.001482 12577 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.000854 12889 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:17.988325  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:17.963549 13167 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:18.005900  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.011117  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:18.012904  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:18.013001  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:18.013744  8650 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:18.014559  8584 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:18.013770  8579 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
W20260709 13:03:18.014766  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 115547us
I20260709 13:03:18.015234  8697 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.022156  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:18.024778  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.025888 13174 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:18.027860 13174 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:18.028501 13174 consensus_queue.cc:1243] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:03:18.028963 13147 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:18.029415 13147 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.029762 13147 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.031605 13148 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.032368  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.016602 12665 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:18.033566  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.033867 12665 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:18.034152 12896 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:18.034345 12665 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:18.012080 13120 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:18.034716  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.035368  8652 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:18.036157  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.036369 13146 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:17.987182 13151 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.038300  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:18.038938  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:18.048539  8578 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:18.048516  8582 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:18.040452  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:17.941493 13136 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:17.987735 13073 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:17.992681 12547 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.992682 12898 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.049846  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:18.099493  8577 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:18.099772  8580 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:18.051699  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:17.962944 13171 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:03:18.053858  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:18.102923  8575 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:18.037658 13114 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:18.065078 13181 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:03:18.065997  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 77777.8us
I20260709 13:03:18.033876 13043 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:18.097323  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 56981.2us
I20260709 13:03:18.098559  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.110337  8579 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:18.100600  8584 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:18.015811 13167 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:18.048008 13029 tablet_copy_client.cc:538] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:03:18.046043 13146 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:18.106812  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:18.109894 13043 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:17.964977 13122 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.013587 13063 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.106850  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.016903 13137 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.137673  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:18.139179 13158 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:18.139289  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:18.125986 13043 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:18.111495  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 96340.4us
W20260709 13:03:18.110212 13168 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 13:03:18.102187 13073 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.096773 13149 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.147403  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:18.159909 13149 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:18.162983 13149 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:18.163640 13149 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.164501 13149 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.166246 13190 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:18.166262  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:18.149297 13187 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:18.178331 13187 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:18.186932 13180 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:18.187431 13180 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.112598 13167 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:18.188738 13158 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:18.125150 13146 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:18.191459 13177 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:03:18.191702 13075 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.112260 13114 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:18.126461 12899 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.187935  8657 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:18.050264 12497 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.187772 13180 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:18.213985  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.105050 12547 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.217608 13136 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.220340  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.221064  9328 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:18.053875 13151 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.223655  8650 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:18.106225 12898 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.111145 13138 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.099s
W20260709 13:03:18.231266  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 183612us
W20260709 13:03:18.233445  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 215771us
I20260709 13:03:18.234921  8550 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.209894 13190 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.238476  8550 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.242213  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:18.191992 13146 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.211851 13137 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.108414 13115 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.210305 13054 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.183645 13063 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.182380 13147 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.242797  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:18.130900 13183 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.113s
I20260709 13:03:18.249559  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.226172 13123 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:18.272022  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2728) took 1089 ms. Trace:
I20260709 13:03:18.272531  8653 rpcz_store.cc:276] 0709 13:03:17.182728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.187285 (+  4557us) service_pool.cc:224] Handling call
0709 13:03:18.262556 (+1075271us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b54000596d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:18.271951 (+  9395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1664216832}
I20260709 13:03:18.274508  8650 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 5. Current term is 6. Ops: []
I20260709 13:03:18.249462 13075 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:18.255280 12497 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:18.286861 12898 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.265205 12268 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:18.258456 13099 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.288784 13158 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:18.329348 13158 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.330554 12913 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:18.333045  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2726) took 1150 ms. Trace:
W20260709 13:03:18.293519 13160 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.496s	user 0.004s	sys 0.000s
I20260709 13:03:18.333482  8655 rpcz_store.cc:276] 0709 13:03:17.182146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.187204 (+  5058us) service_pool.cc:224] Handling call
0709 13:03:18.332837 (+1145633us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b5400380254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:18.332985 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954476416}
I20260709 13:03:18.252931 13190 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:18.341882  9322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1018) took 1168 ms (client timeout 1511 ms). Trace:
W20260709 13:03:18.342360  9322 rpcz_store.cc:269] 0709 13:03:17.173957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.174103 (+   146us) service_pool.cc:224] Handling call
0709 13:03:18.341645 (+1167542us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b5400551c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:18.341805 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171233024}
I20260709 13:03:18.316629 13181 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 13:03:18.348603 13116 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.350139 13116 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.355051 13151 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.298029 13102 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.357220 13043 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.317783 13165 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.257292 13175 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.393781  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2279) took 1689 ms. Trace:
I20260709 13:03:18.323704  8729 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:18.272025 12901 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.394208  8731 rpcz_store.cc:276] 0709 13:03:16.704558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.706852 (+  2294us) service_pool.cc:224] Handling call
0709 13:03:16.707127 (+   275us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:16.944863 (+237736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.389745 (+1444882us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54004731d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:18.393725 (+  3980us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545023232}
I20260709 13:03:18.342226 13143 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.266158 13146 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.280323 13115 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.407819 13156 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.408946 13156 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:18.286954 13174 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:18.409476 13174 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.409668 13193 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.410504 13193 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.367213 12889 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.400363  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.398495 12577 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.403091 13191 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.412791 13191 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.415761  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:18.409868 13174 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.255762 13137 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.279824 13123 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.419042 13137 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:18.411655 13154 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.633s	user 0.000s	sys 0.002s
I20260709 13:03:18.419642 13123 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.420884  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.421567  8656 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:18.421720 13063 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.319206 13187 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.422672 13187 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:18.423375  8548 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.511s (SENT)
I20260709 13:03:18.423910  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.425666 12547 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.426036 12577 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:18.427820  8651 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:18.429534 13143 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.395565 13043 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.320286 13114 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.436874 12994 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:18.437311 12994 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:18.438165 12994 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.439565 12994 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:18.442502 13149 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.446481 13147 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:18.447077  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.448099 13158 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.448772 13099 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.452279 13107 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.452924  8659 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:18.453514  8659 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:18.454058 13122 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.454283  8659 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.455247  8659 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.457828  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:18.458583 12171 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.429554  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2727) took 1247 ms. Trace:
I20260709 13:03:18.459182  8654 rpcz_store.cc:276] 0709 13:03:17.182436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.182548 (+   112us) service_pool.cc:224] Handling call
0709 13:03:17.182837 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.908972 (+726135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.429309 (+520337us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b540018b0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:18.429497 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667208064}
I20260709 13:03:18.461416  8729 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:18.328614 12268 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.463865  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:18.452790 13029 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
I20260709 13:03:18.467551  8731 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:18.468221  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:18.473078  8727 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.459164 12171 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.473261  8655 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:18.463342 12975 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:18.488603  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.488847 12975 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.492264  8731 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:03:18.493468  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.452890 13107 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:18.497380  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.463366 13191 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.463634 12889 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.470379 12268 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:18.472101 13141 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:18.390031 12665 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.449287 13099 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.521870  8731 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:18.504810  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:18.522787  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:18.523500  8729 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:18.469058 13119 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:18.503089 12171 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:18.524624  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 790) took 1306 ms (client timeout 1567 ms). Trace:
I20260709 13:03:18.466374 13018 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 13:03:18.528152  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:18.509850  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 585190us
I20260709 13:03:18.529752  8730 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:18.530457  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.530975  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.531833  9330 raft_consensus.cc:2410] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:03:18.532124  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:18.512169 13158 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.494366 13135 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 13:03:18.466773 13109 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 13:03:18.578241 13135 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:18.578763 13135 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:18.579849  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:18.579689 13135 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.580981 13135 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:18.517174 12665 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.517818  8652 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:18.519711 13175 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:18.520479 13187 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.525985  9324 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:18.525069  9323 rpcz_store.cc:269] 0709 13:03:17.217796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.218487 (+   691us) service_pool.cc:224] Handling call
0709 13:03:18.524440 (+1305953us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b540037a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:18.524556 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":928500608}
I20260709 13:03:18.533109  9322 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:18.592372  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:18.535187  8656 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:03:18.594519  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:18.535751  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.537278  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.538306  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:18.597430  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 59243.2us
I20260709 13:03:18.534379 13054 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.540508 13018 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:18.540647 13191 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:18.542668 13099 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.543546  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.544317  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.547320 12268 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.535025 13141 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:18.490701 13148 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 13:03:18.524827 13073 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.549501  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.540557  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:18.551138 13150 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.542s	user 0.008s	sys 0.000s
I20260709 13:03:18.551668  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:18.552346 12928 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.685818  8655 raft_consensus.cc:2368] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:18.686151 12928 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.639410 13049 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.554231 12975 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.566075  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.689796 13049 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:18.690810  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 124812us
I20260709 13:03:18.692173 13049 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.640305 12665 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.566838  9329 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:18.692977 12665 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:18.693719 12497 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:18.569224 13149 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.572906  8653 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 13:03:18.695081  8653 pending_rounds.cc:85] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 6
I20260709 13:03:18.695387  8653 pending_rounds.cc:107] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 13:03:18.695429 12665 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.695789  8653 raft_consensus.cc:2889] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:18.695576 13146 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.696657 13146 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.581223  8698 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:18.700693  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 119530us
I20260709 13:03:18.572790 13147 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:18.586850  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:18.587797  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.702909 13147 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.703400 13147 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:18.587628  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:18.704324  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 116794us
I20260709 13:03:18.601599  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:18.601842  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.706003  8582 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.603546  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.614799 13137 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:18.603819  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.610049  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.708146  8578 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:18.595285  8656 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:18.615216  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:18.709914  8725 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:18.615718  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.615762  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:18.616369  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.713426  8726 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:18.714790  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.715919  8726 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.617450  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.716897  8728 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.616019  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.717880  8733 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:18.623097  8580 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:03:18.629777  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.719221  8581 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:18.631620 13107 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.614193 13018 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:18.647151  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.657281  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.750128  8575 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.658823  8732 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:18.661653  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.751621  8583 raft_consensus.cc:2410] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:03:18.581914 12913 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:18.658450  9328 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:18.508463 13174 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.681529  9330 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.639015 13105 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.684154 13141 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.685454 12268 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.553834  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:18.573549 12994 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:18.708253  9322 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:18.612262  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.658298 13109 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.727662  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.634438 13054 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.728633  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:18.796762  8656 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:18.729846  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.730312  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:18.802997  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.804275  8728 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:03:18.734542  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.807595  8726 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.735357 13151 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:18.818249  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:18.819056  9328 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:18.665102 13175 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.657526 13187 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.743275  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:18.831472  8727 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:18.746748  9327 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:18.743877  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.833725  8733 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:18.746059  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:18.835836  8656 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
W20260709 13:03:18.836333  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 90407.3us
I20260709 13:03:18.753047  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:18.754060  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50126us
I20260709 13:03:18.755300  8651 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:18.755848  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:18.661242 12577 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:18.760859  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 385705us
W20260709 13:03:18.764983  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.744553 12497 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:18.772611  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:18.771937  9326 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.022s	sys 0.026s
W20260709 13:03:18.846156  9326 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.022s	sys 0.026s
I20260709 13:03:18.846812  8730 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.846284  8729 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.849007  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:18.787710 12969 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.762529 13148 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.068s
I20260709 13:03:18.709661 13137 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.708441 13146 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.809607  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.870759  9322 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:03:18.815243  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 119115us
I20260709 13:03:18.815878  9323 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:18.817068  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.739539 12901 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:03:18.821537  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.771239 13105 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:18.755951 12913 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:18.708877 13150 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.830991  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:18.762720 13107 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260709 13:03:18.884948  8726 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.821817  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:18.876791  8732 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:18.883638  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.887835  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.889150  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:18.809015 12994 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.855347 13043 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:18.819725 13109 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.854866 13151 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:18.872186 13137 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:18.875214 12901 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:18.904361  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.013s	sys 0.017s
W20260709 13:03:18.908643  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.014s	sys 0.017s
W20260709 13:03:18.909376  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 675047us
I20260709 13:03:18.898087  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:18.846122 13175 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.884392 12913 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:18.915827 13150 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.916787 13151 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.917723 13105 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:18.919919 12994 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.854565 12577 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.922367 13109 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.897013 13043 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.923442 13054 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.923767 13043 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:18.924535 13175 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.935211  9328 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:18.894505 13138 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:18.872552 12497 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:18.870343 12969 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.897161 13194 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:18.851068 13187 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.850394 13018 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:18.952679 12171 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:18.963112 12171 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:18.904448 12899 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 13:03:18.923439 12577 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:18.965672 13196 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:03:18.966171 13190 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:18.967231 13018 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.979488 12969 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:18.957242 13194 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:18.990167 13194 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:18.949818 13158 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:18.962327 13156 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:18.945358 13054 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.058s
W20260709 13:03:19.002064 13167 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.779s	user 0.007s	sys 0.003s
I20260709 13:03:18.998027 13174 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.003113 13156 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:19.004767 13191 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:18.999752 13187 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.000097 13158 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:19.049762 13158 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.051563 13158 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:19.036375 13167 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:19.072925 13195 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.130s	user 0.002s	sys 0.000s
I20260709 13:03:19.074069 13198 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:19.079617 13198 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:19.080873 13198 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.063863 13156 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.083972 13156 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:19.090375 13167 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.082589 13198 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:19.092700 13180 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.879s	user 0.007s	sys 0.004s
I20260709 13:03:19.105441 13180 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.101210 13194 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.139055  8577 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:19.141281  8579 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:18.778782  8584 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:19.144819  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.839799  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:18.793035  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:19.147099  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:19.148166  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.148041  8583 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:19.148869  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 308995us
W20260709 13:03:19.152489  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:19.153242  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.153017  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:19.154112  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.155011  8650 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:19.155308  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 714) took 1173 ms. Trace:
I20260709 13:03:19.156292  8584 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:19.155849  8657 rpcz_store.cc:276] 0709 13:03:17.981928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.218010 (+236082us) service_pool.cc:224] Handling call
0709 13:03:18.218359 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:18.219054 (+   695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.219083 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:18.219528 (+   445us) log.cc:844] Serialized 26 byte log entry
0709 13:03:18.478040 (+258512us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:18.478099 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.478207 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.154618 (+676411us) raft_consensus.cc:1685] finished
0709 13:03:19.154675 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.155029 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":405,"raft.queue_time_us":86,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":8046,"thread_start_us":218593,"threads_started":1,"wal-append.queue_time_us":575136}
I20260709 13:03:18.747324  8659 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:03:19.156320  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.764325  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.778155  8725 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:19.163513  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:18.830585  8698 leader_election.cc:400] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:19.165935  8698 leader_election.cc:403] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:03:19.161317  8583 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:19.169363  8582 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:19.170791  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:19.171765  8582 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:19.173025  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:19.174499  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.838964  8652 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:19.178221  8654 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:19.180053  8724 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.004s	sys 0.020s
W20260709 13:03:19.229964  8724 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.004s	sys 0.020s
I20260709 13:03:18.910604  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:19.225358  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:19.226233 12547 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:19.227038  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.252414  8584 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:18.963927 12171 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.011608 13099 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:03:19.293459  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44152.5us
I20260709 13:03:19.295104  8580 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:19.300872 13183 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.301972 12963 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.302083 13135 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.302932 12963 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.303148 13135 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.305534 13063 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:19.306286 13119 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:19.307401 13119 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.300957 12889 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:19.313262 13141 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:03:19.314553  8581 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.595s	user 0.015s	sys 0.003s
I20260709 13:03:19.315068  8581 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:19.318148  8582 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:19.319317 13125 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.319476 13167 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:19.296481  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 465977us
I20260709 13:03:19.229993 13147 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.309540 13148 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.329625  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:19.330334  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.330428 13099 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:19.333760 13099 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.335237 13099 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:19.335727  8579 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:19.339263  8576 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 13:03:19.343809 12928 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.657s	user 0.008s	sys 0.002s
I20260709 13:03:19.342756  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.346712  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.347693  8582 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 6.
W20260709 13:03:19.349680  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.319304 12268 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:19.350993  8582 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:19.352568  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:19.362282  8575 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.611s	user 0.005s	sys 0.010s
I20260709 13:03:19.320513 13125 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.301963 13183 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.376560  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:19.373637  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 220929us
W20260709 13:03:19.382206  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 52660.2us
I20260709 13:03:19.377748  8575 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:19.389798  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63872.6us
I20260709 13:03:19.389580 13167 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.392354 13167 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.393204 12969 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:19.393896  8583 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:19.394542 12969 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:19.395046 12969 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:19.320660 13109 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.112s
I20260709 13:03:19.396240 13109 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:19.396656 13109 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:19.396510 12969 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.402601 13194 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.402737 13180 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.399755  8576 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:19.405123 13193 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.398027 13119 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:19.395176  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.397735 13135 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.405779 13220 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:19.408522 13109 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.408416  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:19.397792  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 487151us
I20260709 13:03:19.379146 12963 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.418596  8581 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260709 13:03:19.418874  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 706) took 1445 ms (client timeout 1631 ms). Trace:
W20260709 13:03:19.420045  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 521841us
I20260709 13:03:19.425832 12497 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.426965  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:19.427718  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51318.4us
I20260709 13:03:19.428233 12975 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:18.993696 12901 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.411016  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:19.440150  8584 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:19.443704  8658 rpcz_store.cc:269] 0709 13:03:17.972974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.988140 (+ 15166us) service_pool.cc:224] Handling call
0709 13:03:19.418456 (+1430316us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400420c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:19.418669 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2735601920}
I20260709 13:03:19.466682  8580 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:19.376629 13114 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260709 13:03:19.450541 12975 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:19.481104 13138 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:19.481750 12975 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:19.471612  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:19.483731  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.485917  8579 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:19.487299  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.488305  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.489985  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.490981  9329 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:19.491022  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:19.492128  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:19.492667  9329 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:19.492938  9327 raft_consensus.cc:2376] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9. Re-sending same reply.
I20260709 13:03:19.384740 13148 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.490091 13183 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.472277  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.493068  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:19.494524  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.489123  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.485172  8584 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:19.490264  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.483726  8576 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:19.497848  8582 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
W20260709 13:03:19.499312  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 88458.5us
I20260709 13:03:19.501024 12268 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:19.501677 12268 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:19.503069 12268 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.503862  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:19.504762  8650 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:19.506325 13180 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:19.508814  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.510056  9328 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:19.511773 12547 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:19.516139  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.516995  8584 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
W20260709 13:03:19.517247  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 197498us
I20260709 13:03:19.520424  8579 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:03:19.521180  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44403.4us
I20260709 13:03:19.521752  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.524221 13183 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:19.524750 13183 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:19.525228 13183 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:19.527344  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:19.530439  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.532424  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:19.520879  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 94237.1us
I20260709 13:03:19.536370  8580 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:19.537117  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.538715 13054 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:19.538905  8581 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:19.539831  8584 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 13:03:19.540473 13147 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.541261  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.538772  8576 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:19.543288  8576 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: [5.5-9.7]
I20260709 13:03:19.543201  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:19.544358 13193 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.544905 13202 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:19.545037 13193 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.545703 13202 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:19.545979  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50393.6us
I20260709 13:03:19.546152  9328 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:03:19.546662  9328 pending_rounds.cc:85] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 8
I20260709 13:03:19.546976  9328 pending_rounds.cc:107] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:03:19.547068  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:19.547866 13193 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:19.548854  8578 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.797s	user 0.004s	sys 0.012s
I20260709 13:03:19.549422  8578 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:19.547364  9328 raft_consensus.cc:2889] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:19.493518 13123 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.118s
I20260709 13:03:19.551303  8580 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: [9.6-11.7]
I20260709 13:03:19.549831  8584 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
W20260709 13:03:19.551211  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 58207.5us
I20260709 13:03:19.541796  9327 raft_consensus.cc:2368] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:19.544447 12975 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:19.527544 13180 raft_consensus.cc:2764] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:03:19.524240 12901 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:03:19.560213  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.562072 12969 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:19.571058  8581 raft_consensus.cc:1217] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Deduplicated request from leader. Original: 3.3->[4.4-11.6]   Dedup: 4.4->[10.5-11.6]
I20260709 13:03:19.571607  8581 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: [4.4-11.6]
I20260709 13:03:19.575582  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:19.576054  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.577994  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:19.578788  9328 raft_consensus.cc:2376] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10. Re-sending same reply.
I20260709 13:03:19.580173  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 785) took 2372 ms. Trace:
I20260709 13:03:19.584905  8578 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: [7.6-9.7]
I20260709 13:03:19.586745 13073 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.588462  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:19.589628  8583 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:19.591225  8583 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:19.593321  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.594833  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:19.594547 12899 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:19.570251 13109 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:19.551025 13156 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:19.603678  8578 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: [8.6-10.7]
I20260709 13:03:19.612826  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:19.568161 13220 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 13:03:19.580615  9331 rpcz_store.cc:276] 0709 13:03:17.207188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.208165 (+   977us) service_pool.cc:224] Handling call
0709 13:03:18.017544 (+809379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:18.017857 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.017883 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:18.051176 (+ 33293us) log.cc:844] Serialized 26 byte log entry
0709 13:03:18.588250 (+537074us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:18.588311 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.588405 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.072107 (+483702us) spinlock_profiling.cc:243] Waited 64 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:19.259257 (+187150us) raft_consensus.cc:1685] finished
0709 13:03:19.267046 (+  7789us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.579745 (+312699us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b5400542750. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:19.579950 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":46856,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":150,"spinlock_wait_cycles":2007766016,"thread_start_us":477842,"threads_started":1,"wal-append.queue_time_us":770149}
I20260709 13:03:19.623528  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.624796  8650 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:19.625962 13043 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.682s	user 0.009s	sys 0.008s
I20260709 13:03:19.626502 13043 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.628914 13043 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.629794  9324 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:19.633805  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.634753  8658 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:19.547184 13202 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.642732 13220 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:19.643314 13220 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:19.646347  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51189.2us
I20260709 13:03:19.647565  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:19.610738 12889 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:19.623133  8578 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:19.644492 13220 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.655207  8583 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:03:19.642575 13125 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.657832 13180 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:19.658344  8658 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 13:03:19.658788  8658 pending_rounds.cc:85] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 8
I20260709 13:03:19.621591  9323 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:19.659063  8658 pending_rounds.cc:107] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 13:03:19.659344  8658 raft_consensus.cc:2889] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:19.652822  9329 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 6. Current term is 7. Ops: []
I20260709 13:03:19.620806  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:19.620126  8622 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.870s (SENT)
I20260709 13:03:19.668386  9330 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:19.669391 13147 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.670298 13147 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.672048  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.672924  8733 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:19.674160 13156 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.674798 13156 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.677865  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.678912 13156 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.678684  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2702) took 1078 ms. Trace:
I20260709 13:03:19.679004  8727 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:19.667313  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:19.682905  8577 rpcz_store.cc:276] 0709 13:03:18.600240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.601194 (+   954us) service_pool.cc:224] Handling call
0709 13:03:19.678329 (+1077135us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b54003a4154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:19.678569 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":994858880}
I20260709 13:03:19.681361  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:19.691232 13073 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:19.691797 13073 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:19.692345 13073 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:19.658346 13180 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:19.693367 13180 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:19.694351  8652 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:19.700515 13063 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:19.700809  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:19.701176 13063 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:19.701905 13191 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:19.703526  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.704952 13147 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.712390  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.712877  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.715881  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.717433  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:19.718282 13212 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.331s	user 0.003s	sys 0.001s
I20260709 13:03:19.691574  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.719951  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1060) took 1164 ms. Trace:
I20260709 13:03:19.720367  9326 rpcz_store.cc:276] 0709 13:03:18.555412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.847390 (+291978us) service_pool.cc:224] Handling call
0709 13:03:18.847811 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:18.854750 (+  6939us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:18.854791 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:18.875412 (+ 20621us) log.cc:844] Serialized 26 byte log entry
0709 13:03:19.379850 (+504438us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:19.427566 (+ 47716us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.427694 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.717505 (+289811us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:19.719435 (+  1930us) raft_consensus.cc:1685] finished
0709 13:03:19.719463 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.719755 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":871,"raft.queue_time_us":105,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":8154,"spinlock_wait_cycles":92624256,"thread_start_us":383910,"threads_started":1,"wal-append.queue_time_us":447511}
W20260709 13:03:19.725821 13137 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.699s	user 0.011s	sys 0.008s
I20260709 13:03:19.726567 13137 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.729650  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:19.730284  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2760) took 1181 ms. Trace:
I20260709 13:03:19.730819 12889 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:19.731377 12889 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:19.731683  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:19.731786 12889 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:19.735203  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1061) took 1049 ms. Trace:
I20260709 13:03:19.750458  8656 raft_consensus.cc:2410] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:03:19.733139  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.741808  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.667954 13148 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.733484 13114 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:03:19.774768 13191 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:19.775281 13114 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:19.775857 13114 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:19.776954  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:19.777771  8732 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:19.777623  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:19.777967 13114 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.775372 13191 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:19.779440 13114 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:19.779536 13191 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:19.780493 13167 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.781100 13167 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.783972 13167 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:19.776975  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.785888  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.786633  8730 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:19.775792  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.790505  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.791692  8650 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:03:19.793630  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:19.795847 12975 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:19.733203 12963 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.063s
W20260709 13:03:19.764788  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.729280 13137 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:19.740342 13220 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:19.793135  8732 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:19.753696  9322 rpcz_store.cc:276] 0709 13:03:18.555742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.877889 (+322147us) service_pool.cc:224] Handling call
0709 13:03:18.878191 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:18.878382 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:19.320383 (+442001us) log.cc:844] Serialized 14 byte log entry
0709 13:03:19.537841 (+217458us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:19.565326 (+ 27485us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:19.574817 (+  9491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.592324 (+ 17507us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.605345 (+ 13021us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35066,"prepare.queue_time_us":186997,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":21207,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":176,"thread_start_us":577811,"threads_started":2,"wal-append.queue_time_us":300021}
I20260709 13:03:19.752629 13073 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.804586 13073 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:19.805815  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 158400us
I20260709 13:03:19.730856  8654 rpcz_store.cc:276] 0709 13:03:18.549007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.549190 (+   183us) service_pool.cc:224] Handling call
0709 13:03:19.730013 (+1180823us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54001d1fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:19.730212 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1139867904}
I20260709 13:03:19.812613  8575 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:19.823483  8729 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
W20260709 13:03:19.751236 13219 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.347s	user 0.003s	sys 0.000s
W20260709 13:03:19.835505  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:19.837682  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.739555 12171 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:19.740935 13147 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:19.738029 13198 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.632s	user 0.007s	sys 0.006s
W20260709 13:03:19.848654  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 113645us
I20260709 13:03:19.854364  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.855355  8726 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:03:19.856194  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.856534  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.857317  8653 raft_consensus.cc:2410] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:03:19.858229  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.859174  8653 raft_consensus.cc:2410] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:03:19.861778  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.862486  8651 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:19.863459  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:19.864290  8658 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:19.875054 13183 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:19.877579 12913 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.807s	user 0.013s	sys 0.013s
I20260709 13:03:19.878284 12913 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.885850 13158 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:03:19.886850  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45360.2us
I20260709 13:03:19.887599 12889 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.877651  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:19.891844 13167 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:19.892288 13167 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:19.892632 13167 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:19.888463 13099 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:03:19.897485  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45578.2us
I20260709 13:03:19.885102  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:19.898895  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:19.899585  8581 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:19.849380  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:19.900070  8724 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:19.909584  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.909832  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:19.910403  8650 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:19.910507  9322 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:19.910961  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:19.891407 13180 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.912055  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:19.913008 13073 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:19.849773 13137 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.888762  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:19.901216  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:19.913908  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 297098us
I20260709 13:03:19.914048  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:19.915639  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.941756  8724 raft_consensus.cc:2410] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:03:19.917635  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.914687  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:19.918048  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.918681  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.947531  8733 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
W20260709 13:03:19.949293  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.929996  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.930729  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:19.934181  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45598.1us
I20260709 13:03:19.936542  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.915107  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:19.927457  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 78130.3us
I20260709 13:03:19.955358  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:19.914953 13224 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
I20260709 13:03:19.944607  8727 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:19.928848  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.966773  8656 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:03:19.934595  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2572) took 1315 ms. Trace:
I20260709 13:03:19.929185  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:19.914289  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2099) took 1217 ms. Trace:
W20260709 13:03:19.921505  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.957585  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.944672  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:19.920488  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:19.935698  8697 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.575s (SENT)
I20260709 13:03:19.930133  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.931105  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:19.929565  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.927802  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.924422  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:19.973325  8655 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:03:19.920207 13158 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.913571 13073 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:19.974373  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:19.913461 13180 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:19.976289  8650 raft_consensus.cc:2410] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:03:19.930315 13043 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:20.020673  8726 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:19.923066 12889 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:19.934517 12913 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:19.934412 13099 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:19.918265 13141 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.118s
I20260709 13:03:19.967449  8728 rpcz_store.cc:276] 0709 13:03:18.619168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.829855 (+210687us) service_pool.cc:224] Handling call
0709 13:03:18.831088 (+  1233us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:18.855262 (+ 24174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.855298 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:18.870963 (+ 15665us) log.cc:844] Serialized 26 byte log entry
0709 13:03:19.025481 (+154518us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b480018c258. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:19.348643 (+323162us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:19.367135 (+ 18492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.384443 (+ 17308us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.750277 (+365834us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:19.934193 (+183916us) raft_consensus.cc:1685] finished
0709 13:03:19.934268 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.934408 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23743,"raft.queue_time_us":29482,"raft.run_cpu_time_us":2766,"raft.run_wall_time_us":155770,"spinlock_wait_cycles":73704064,"thread_start_us":119333,"threads_started":1,"wal-append.queue_time_us":518786}
I20260709 13:03:20.046440  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:20.047581  8650 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:19.936069 13063 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.918067 13137 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:19.928435  8479 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 10, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 10 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:19.935298 13183 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.022481  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.053372 13063 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.055799 13137 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.057261  8479 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:20.067378  8479 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:20.075556  8479 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 8, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 8 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:19.927615 13125 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:19.926801 13198 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:20.026420  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:20.026494  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 59829.1us
I20260709 13:03:19.967658  8576 rpcz_store.cc:276] 0709 13:03:18.697086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.547535 (+850449us) service_pool.cc:224] Handling call
0709 13:03:19.548010 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:19.548614 (+   604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:19.548647 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:19.549469 (+   822us) log.cc:844] Serialized 26 byte log entry
0709 13:03:19.689615 (+140146us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:19.689653 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.689729 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.913970 (+224241us) raft_consensus.cc:1685] finished
0709 13:03:19.914022 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.914212 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5485,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":44958,"thread_start_us":139669,"threads_started":1,"wal-append.queue_time_us":209087}
I20260709 13:03:19.933497 12901 raft_consensus.cc:2764] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:03:20.027559  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:20.028991  8478 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 303) took 39024 ms (client timeout 15000 ms). Trace:
I20260709 13:03:20.029711  8654 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:20.030306  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.108530  8656 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:20.031124  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.033183  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:20.057497  8733 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:20.059557  9324 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:20.060020  8653 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:19.978686 13073 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:19.980309  8659 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:19.993389  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:20.019874 13158 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:19.935380 13191 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:20.090991  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 158767us
I20260709 13:03:20.117048  8658 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:20.090991  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 116739us
I20260709 13:03:20.048970  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:20.123349  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 178773us
I20260709 13:03:20.113137  9322 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:20.117012 12171 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:19.974304 13220 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:20.125425  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:20.117210  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 140844us
I20260709 13:03:20.129544  8726 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:20.132346  8650 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:03:20.134725 13099 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:03:20.134992  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 199483us
I20260709 13:03:20.135876 13099 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:20.107714  8478 rpcz_store.cc:269] 0709 13:02:40.996547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.996898 (+   351us) service_pool.cc:224] Handling call
0709 13:03:20.020987 (+39024089us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:11.947088 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:12.073921 (+126833us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:12.077885 (+  3964us) write_op.cc:435] Acquired schema lock
0709 13:03:12.080156 (+  2271us) tablet.cc:662] Decoding operations
0709 13:03:12.242335 (+162179us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:12.259489 (+ 17154us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:12.261808 (+  2319us) tablet.cc:685] Acquiring locks for 13 operations
0709 13:03:12.701821 (+440013us) tablet.cc:701] Row locks acquired
0709 13:03:12.701869 (+    48us) write_op.cc:260] PREPARE: finished
0709 13:03:12.702046 (+   177us) write_op.cc:270] Start()
0709 13:03:12.702221 (+   175us) write_op.cc:276] Timestamp: P: 1783602192701985 usec, L: 0
0709 13:03:12.702253 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 13:03:12.703059 (+   806us) log.cc:844] Serialized 5095 byte log entry
0709 13:03:13.627371 (+924312us) op_driver.cc:464] REPLICATION: finished
0709 13:03:13.825764 (+198393us) write_op.cc:301] APPLY: starting
0709 13:03:13.825911 (+   147us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:15.082825 (+1256914us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:15.082982 (+   157us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:15.221317 (+138335us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:15.319621 (+ 98304us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 13:03:15.326605 (+  6984us) write_op.cc:312] APPLY: finished
0709 13:03:15.349751 (+ 23146us) log.cc:844] Serialized 121 byte log entry
0709 13:03:15.538578 (+188827us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:15.538770 (+   192us) write_op.cc:454] Released schema lock
0709 13:03:15.549091 (+ 10321us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:15.611595 (+ 62504us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:15.611628 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6643510,"reported_tablets":40,"child_traces":[["op",{"apply.queue_time_us":198009,"apply.run_cpu_time_us":49096,"apply.run_wall_time_us":1785819,"cfile_cache_hit":50,"cfile_cache_hit_bytes":52225,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6216,"cfile_init":2,"lbm_read_time_us":58908,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":10,"num_ops":13,"prepare.queue_time_us":892949,"prepare.run_cpu_time_us":73830,"prepare.run_wall_time_us":935343,"raft.queue_time_us":513416,"raft.run_cpu_time_us":109404,"raft.run_wall_time_us":234763,"replication_time_us":896922,"thread_start_us":1546745,"threads_started":5,"wal-append.queue_time_us":433509,"wal-append.run_cpu_time_us":48055,"wal-append.run_wall_time_us":2418638}]]}
I20260709 13:03:20.136533 13099 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.137704 13220 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.138145 13220 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:20.138818 13063 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.139402 13063 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.139792 13099 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:20.135993  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:20.145390  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 253944us
I20260709 13:03:20.106150 13198 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.107694 13125 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.147755  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:20.148547  9324 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:20.149085  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:20.151958  8658 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:20.152333  8654 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:20.154556 13198 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:20.114670 12913 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:20.160912  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:20.156373  8657 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:20.157200  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:19.923776 13138 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.193s
I20260709 13:03:20.165189  9322 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:20.143329 12889 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:20.170502  8731 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:20.171350  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.178489  8659 raft_consensus.cc:2410] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:03:20.174553  8479 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 9, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 9 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:20.154037 13125 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:20.182171  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:20.165030 12913 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.171171 12889 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.186851  8479 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 3 to 5, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 5 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:20.189528  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58000.1us
I20260709 13:03:20.191183  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:20.195866  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.193151 12889 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.203908 12969 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:20.204552 12969 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:20.205582 12969 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:20.206749  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60455.2us
I20260709 13:03:20.207947  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:20.209757 13156 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:20.210605 13156 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.211000 12913 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.215935  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:20.206684 13158 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.220067 13158 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:03:20.220568 13158 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.220983 13158 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:20.222425  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.227391 13167 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.230278 13180 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:20.230786 13180 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.231181 13180 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:20.242390 13063 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:20.246830  8656 raft_consensus.cc:2410] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
W20260709 13:03:20.248770  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 77785.5us
W20260709 13:03:20.250445  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:20.251560  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 100950us
I20260709 13:03:20.258930  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.228906 12889 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:20.270489  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 96247.6us
I20260709 13:03:20.281701  8479 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:20.279219 12889 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:20.286072 13191 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:20.286383  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 115100us
I20260709 13:03:20.287935 12963 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:20.288415 12963 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.288810 12963 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:20.289572 12171 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:20.290887  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:20.291828  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 170628us
I20260709 13:03:20.292608  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.293594 13183 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:20.294230 13183 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:20.295163 13099 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:20.296368 13073 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.257056 13141 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:20.297778 13141 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.298238 13141 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.298830 13018 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:20.302119  8479 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:20.306913  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:20.310937  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.311517  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.312239  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.259186 12913 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:20.277004 13220 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.319023  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.323621  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:20.318059  8479 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:20.328876  8653 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.330369  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.334004  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:20.339897  9322 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:20.262661 13148 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 13:03:20.347828  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:20.350189  8654 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:20.345842  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:20.361467  9327 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:20.363600  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45184.2us
I20260709 13:03:20.364612  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:20.366977 13191 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:20.332671 13167 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.358148  8479 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 6 to 7, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 7 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:20.359980 13147 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.514s	user 0.012s	sys 0.000s
I20260709 13:03:20.371503  8659 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:20.367585 13156 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:20.390719  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44997.6us
I20260709 13:03:20.372063 12913 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:20.393508  8656 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
W20260709 13:03:20.396823  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 79634us
I20260709 13:03:20.378975 12889 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:20.400070  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.381605 13147 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:20.408891  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.414083 13156 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:20.414616 13156 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:20.415010 13156 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:20.417798 12975 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.418440 12975 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.420565 12975 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.395035 12913 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.421351 12913 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:20.426805 13233 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 13:03:20.429602  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2092) took 1734 ms. Trace:
I20260709 13:03:20.430136  8579 rpcz_store.cc:276] 0709 13:03:18.695418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.521205 (+825787us) service_pool.cc:224] Handling call
0709 13:03:19.682024 (+160819us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:19.691703 (+  9679us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:19.691736 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:19.692626 (+   890us) log.cc:844] Serialized 52 byte log entry
0709 13:03:19.995754 (+303128us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:20.369845 (+374091us) log.cc:844] Serialized 14 byte log entry
0709 13:03:20.370203 (+   358us) log.cc:844] Serialized 14 byte log entry
0709 13:03:20.377754 (+  7551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.377891 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:20.377932 (+    41us) raft_consensus.cc:1685] finished
0709 13:03:20.377945 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.427097 (+ 49152us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54003a4150. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:20.429528 (+  2431us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4593,"raft.queue_time_us":37906,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":15327,"spinlock_wait_cycles":378275456,"thread_start_us":626533,"threads_started":2,"wal-append.queue_time_us":460847}
I20260709 13:03:20.379663 13114 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
W20260709 13:03:20.441627  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 154164us
I20260709 13:03:20.432459 13156 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.412002  8479 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 6 to 9, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 9 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:20.440951  8653 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:20.441973  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:20.452522  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 71196.1us
I20260709 13:03:20.401122 12889 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:20.456053  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:20.455410  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 260843us
I20260709 13:03:20.463167  8479 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:03:20.465946  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65438us
I20260709 13:03:20.473788  9322 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:03:20.442425 13167 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:20.481683  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:20.483572  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.484169  9322 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:20.458909 12889 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:20.505517  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:20.487010  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 85200.4us
I20260709 13:03:20.507498  9329 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:20.486927 12171 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.511025  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:20.511852  9326 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.512696  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:20.513899  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:20.516016  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 192804us
I20260709 13:03:20.516295  8479 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 5, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 5 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:20.511085  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:20.483001 13147 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:20.530237 12171 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.515429 13114 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:20.531633 13167 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:20.511783 13073 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.545437  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:20.543761  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 93418.9us
I20260709 13:03:20.546588  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:20.549939 12171 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.539075 13191 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:20.550913  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:20.550730  8479 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:20.551695  8581 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:20.552006 13191 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.547197 13147 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:20.554387 13147 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.530257 12975 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:20.524124 13220 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.552456  8479 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:20.560783 12268 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.567464  8576 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:20.569084  8579 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:20.567750 12268 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:20.571205  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104575us
I20260709 13:03:20.511607 13180 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.578557  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.578250  8479 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 7, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 7 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:20.585602 12171 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.544827 13114 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:20.586429 12171 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.593758  9328 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:20.596042  8579 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:03:20.575675 13018 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.603920 13018 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.609812  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.609761 13158 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.610273  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:20.612102 13158 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.612988  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.613765  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.614354  8656 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.609958 12268 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.609067  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:20.621966  8653 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:20.602443 13167 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:20.626791  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:20.623162 13156 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.632027  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:20.632728  8657 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:20.638054  8579 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:03:20.644855  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.646217 12913 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.646996  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.648499 12913 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.649508  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.658044  8653 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:20.627333  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:20.659921  8659 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:20.666723  8698 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.616670 13247 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 13:03:20.668639 13255 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:20.704290 13109 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.669930  9330 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:20.722182 13109 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.702673 13073 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:20.724145 12913 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:20.724520 13109 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.670064 13180 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:20.724449 13073 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:20.725914 13250 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260709 13:03:20.726688  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 79798.9us
I20260709 13:03:20.727495  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.727816 13073 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:20.673275  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 68987.4us
I20260709 13:03:20.728967  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:20.679818  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:20.680576  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:20.741576  9322 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:20.681494  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:20.685086  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:20.685743  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:20.737140  9326 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:20.744834  8580 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:20.686429  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:20.687071  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:20.687753  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:20.691283 12171 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.691956  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.693089 13018 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.746367  8577 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 6.
I20260709 13:03:20.699277  8657 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:20.699167  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:20.705628 13220 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.703681  8652 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:20.706993  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.708043  8656 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:20.708120  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:20.768513  8579 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:20.710742  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:20.769807  9331 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
W20260709 13:03:20.713641  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47219us
I20260709 13:03:20.727975 13180 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.729251  8697 leader_election.cc:400] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:20.670647  9325 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:20.729892 12901 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:20.732290 13125 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.533s	user 0.009s	sys 0.000s
W20260709 13:03:20.775413  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 184035us
I20260709 13:03:20.778241  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:20.778869  9326 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:20.702412 13183 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 13:03:20.674552  9329 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:20.734242 13240 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:20.735886 12913 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.706617 13167 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 13:03:20.697880 13236 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:20.700861 13242 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:03:20.697412  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:20.700363  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:20.765255  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:20.771977  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65073us
I20260709 13:03:20.766371 13246 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.799573  8656 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:03:20.800352  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.800844  8584 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:20.773037  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44155.8us
I20260709 13:03:20.766602  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:20.746030 12963 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.775774 13018 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:20.777113 13029 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=1 applied=7 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 13:03:20.767374  9297 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.657s (SENT) [suppressed 25 similar messages]
I20260709 13:03:20.670167 13114 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.789242  9322 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
W20260709 13:03:20.790135  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98240.2us
I20260709 13:03:20.790064 12901 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.782019 13248 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:03:20.793004 13198 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.637s	user 0.005s	sys 0.008s
I20260709 13:03:20.814096 12171 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:20.815796  8580 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:20.790088 12913 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.784086 13180 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.789947 13125 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:20.820581  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 91581.8us
I20260709 13:03:20.822165  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.823784  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:20.826474  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.811816 13018 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.836613  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:20.778350 13220 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.815505  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:20.861964  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:20.842247  9326 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
W20260709 13:03:20.820343  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 120189us
I20260709 13:03:20.843981 13156 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.845636  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:20.866626  8724 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:20.847620 13138 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:03:20.848029 13262 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.084s	user 0.003s	sys 0.000s
I20260709 13:03:20.849545 13029 tablet_bootstrap.cc:492] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:03:20.837069 13236 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:20.837232 13198 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.845491 13114 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.827126 13255 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:03:20.859490  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 92157.7us
I20260709 13:03:20.825412 13248 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:20.830869 13183 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:20.866964 13191 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:20.868271  8731 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:20.866223 13043 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:20.875442  8729 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:20.837533 13073 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
W20260709 13:03:20.878787  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 56707.7us
I20260709 13:03:20.860513 13018 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.869257 13236 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:20.813725 13246 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.834759 12889 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.878114  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:20.878717  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:20.986665  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 108061us
I20260709 13:03:20.871064 13156 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:20.867602  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:20.886001  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:20.986093  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 108055us
I20260709 13:03:20.990772  8724 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:20.893123  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:20.992044 12268 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:20.956777  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:20.884990 13255 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:20.979406 13236 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:20.981101 13148 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.981190 13220 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.981673 12171 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.022881  8652 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:03:20.982419  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:20.982574  8656 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:03:20.982396  9327 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:20.983987 12889 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:20.888577 13043 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:20.985224 13180 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:20.882208 13183 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.045338  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:20.885788 13235 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:20.882436 13248 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:20.892081 13029 ts_tablet_manager.cc:1403] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 2.440s	user 0.057s	sys 0.042s
I20260709 13:03:20.994238  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:20.995905  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.050091  8580 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:21.050408  8653 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 13:03:20.996690  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 134783us
I20260709 13:03:20.997284  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.034041  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.000583 13147 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:21.036896  8657 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 7: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:21.023034 12171 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.039572  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:21.000926 13141 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.694s	user 0.013s	sys 0.000s
I20260709 13:03:21.046717  8733 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:21.048681 13183 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:21.034688  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 234362us
I20260709 13:03:21.055276  8659 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:21.065416  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:21.055248  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56193.2us
W20260709 13:03:21.059047  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64928us
I20260709 13:03:21.000367 13158 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:21.060099  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 198983us
I20260709 13:03:21.059294  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.059096  8730 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:21.041791 13043 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.042387 13099 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:21.066497 13148 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.054625 13236 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.064177 13141 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.055415 13063 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:03:21.057235 12901 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:21.080358  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:21.058434 13180 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:21.096208 13099 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:21.097244 12171 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.088768  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.067536 13242 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:21.094237  8625 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.073581 12975 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:21.086923 13158 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.066766 13147 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:21.122308  8658 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:21.068116 13276 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:21.101007 13180 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:21.107137  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 109914us
I20260709 13:03:21.109166 12171 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:21.127741 12171 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.128123 12171 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:21.125059 13137 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:21.128661 13137 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:21.061091 13247 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 13:03:21.122946 13158 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.109712  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:21.095757 13073 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:21.128571  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.130652 12268 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:21.131129 12268 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:21.076962 13018 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.130721  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.133514 12268 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.105481 12913 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.140204  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:21.137704  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:21.141517  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 141947us
I20260709 13:03:21.123502 12963 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:21.141326  8657 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:21.130085 13125 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.144708  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:21.119163 13278 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:21.146571  8725 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:21.153512 13277 raft_consensus.cc:2764] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260709 13:03:21.154099  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 83646.1us
I20260709 13:03:21.153865 13180 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.145715 13073 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.160347 12913 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.146972 13240 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:21.131714 13255 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:21.131995 13274 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:21.166483 13264 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.402s	user 0.003s	sys 0.000s
I20260709 13:03:21.169812 13147 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.176067  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.133742 13276 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:21.189818 13278 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:21.189643 13158 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.191341 13018 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.191505 13180 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:21.134581 13137 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.193224 13073 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.203080  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:21.205814 12171 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:21.208021  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 77363.9us
I20260709 13:03:21.222198  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.142895 13198 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:21.220260  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:21.223251  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.223665  9322 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:21.225023  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 115231us
I20260709 13:03:21.225941  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:21.226727  9322 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.226809  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.228091 13137 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.229039 13240 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:21.228761 13276 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.229288 13158 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:21.229775  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.230108  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:21.230998 13261 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260709 13:03:21.231264 13274 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:21.218809 13220 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:03:21.236378 13029 raft_consensus.cc:359] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:21.238722 12171 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.240154  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:21.272903  9329 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.241206  9322 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:21.241655 13281 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:03:21.252231  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 61659.2us
I20260709 13:03:21.254681  8655 raft_consensus.cc:2368] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
I20260709 13:03:21.218653 13278 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:21.258136  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.260596 13198 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:21.262128 13191 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:21.262629 13043 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.263888 13148 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:21.282351  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52336.2us
I20260709 13:03:21.267493  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:21.280031  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.275394 13029 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:03:21.287732  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:21.276976 13278 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.279883 13191 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.276034 13220 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:21.275426 13236 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:21.279294 13281 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:21.275934 13248 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.306054 13281 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.307065 13248 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:21.307935  9328 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:21.308610 13246 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.295480 13286 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:21.309732 13286 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.310174 13286 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:21.312338  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 223613us
I20260709 13:03:21.314023  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:21.280231 13240 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:21.314568  9329 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:21.296563 13043 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.287242 13198 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:21.287333 13148 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:21.313897 12889 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:21.319322  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.320643 12975 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 13:03:21.321988 12913 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:03:21.322326  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 119344us
I20260709 13:03:21.323395  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:21.323743 12889 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:21.325280 12268 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.324828 12889 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.325670 12969 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:21.326128 12969 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:21.327024 12969 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:21.327732 13269 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.455s	user 0.003s	sys 0.000s
I20260709 13:03:21.293838 13147 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:21.323386 12913 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:21.333387 13114 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:21.340899 13220 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.341708 13245 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260709 13:03:21.328405 13099 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:21.342882 13245 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:03:21.343259 13245 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:21.345681  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.350083 13248 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:21.348057 13063 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 13:03:21.349601  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:21.351783 13114 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.328694 13246 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:21.321182 13240 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.356932  9325 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:03:21.352754 13043 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.005s
I20260709 13:03:21.344141 13245 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.356932  9323 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:03:21.358726 13245 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:21.358908 13240 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.335044 13274 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.334969 13147 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.333521 13148 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:21.344812 12969 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.319116 13183 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.372393 13183 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.343211 13099 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.334177 13029 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:21.388079 13029 ts_tablet_manager.cc:1434] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.327s	user 0.007s	sys 0.000s
I20260709 13:03:21.350260 13198 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:21.392592  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58682.7us
I20260709 13:03:21.393644  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.394516 13183 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:21.395323  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.396167 13183 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:21.396603  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.397537 13183 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:21.397996  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.398975 13183 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:21.399474  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.400499 13183 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:21.401023 13183 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.415906  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.416452  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.416594  8731 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:21.417205  8729 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:21.418251  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.418927  8728 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:21.419814  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.420395  8727 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.421748  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.421744 13280 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:21.423346 13280 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:21.424144 13265 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.446s	user 0.004s	sys 0.000s
I20260709 13:03:21.424235 13280 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.428275  8625 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.400663  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:21.391323 13247 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:21.347148 13191 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.408444  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:21.367635 13147 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.367882 13246 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:21.441058  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.441507 12171 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:21.441797  8725 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:21.441974 12171 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.442390 12171 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.442472  8727 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:21.438588  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.442579  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.432058  8731 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:21.443660  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.445271  9322 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:21.446270  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.446722  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.446945  8733 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.443807  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.448553 13274 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:21.453282  8732 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:03:21.453715  8732 pending_rounds.cc:85] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:03:21.453936  8732 pending_rounds.cc:107] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:03:21.454145  8732 raft_consensus.cc:2889] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:21.454713  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.455242  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.455896  8732 raft_consensus.cc:2376] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 11. Re-sending same reply.
I20260709 13:03:21.456182 12268 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:21.456506 12268 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:21.457329 12268 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.457763  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:21.458609  9331 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
W20260709 13:03:21.459050 13291 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:03:21.460027 13141 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.460372  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:21.461097 13247 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:21.461488 13240 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:21.462738 13191 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:21.464288 13273 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.590s	user 0.004s	sys 0.000s
W20260709 13:03:21.461135 13292 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:03:21.453115 13246 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.468755 13156 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.461568 13247 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.469666 13247 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.469971  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:21.470571  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.470868 13138 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:21.471169  8726 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.471313 13138 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:21.455351  8724 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:21.472544  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.473280  9322 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:21.473341  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.473912  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.475610  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.425376 13280 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:21.458509 12268 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.456750  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.460382  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.472293 13138 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:21.497051  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50395.3us
W20260709 13:03:21.498631  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.499598  9296 leader_election.cc:400] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
I20260709 13:03:21.504154  9296 leader_election.cc:403] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:21.504446  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.506578  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:21.440668 13183 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.507813  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.508389  8657 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.466109 13286 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.506680  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.509799 12171 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.511041  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.512796  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:21.513868  9328 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:21.442705 12963 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.512861  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:21.514884  9326 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.452057 13158 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:21.451198 13242 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:21.515774  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.529556  8658 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:21.532650  8726 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:21.451668 13137 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:21.527413  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.534902  8650 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:21.534790  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.535646  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.527616  8652 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:21.534920  9331 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:21.533902  8732 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:21.516199  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.542657  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.543676  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.544459  8580 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.544875  8653 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:21.508214 13125 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:21.508929 13240 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:21.554759  8655 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.513906 13063 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
W20260709 13:03:21.567499  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 253535us
I20260709 13:03:21.568768  8724 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:03:21.571142  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.536475 13158 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.572842  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.537151 13242 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.575465  8584 raft_consensus.cc:2376] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12. Re-sending same reply.
I20260709 13:03:21.577044  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.580561 12268 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.581040 12268 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.582444  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.561946 13125 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.554153 13167 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:21.562474 13238 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.596527  9329 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:21.535778 12963 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.583835  8580 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:21.588498  9326 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:21.590479  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.611356  8650 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:21.556567 13137 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:21.591531  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.612392  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:21.613144  8724 raft_consensus.cc:2368] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
I20260709 13:03:21.579025 13158 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.613966  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.614182  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.614636  8575 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9. Re-sending same reply.
I20260709 13:03:21.614898  8581 raft_consensus.cc:2376] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11. Re-sending same reply.
I20260709 13:03:21.592285  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.593945  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.594655 12268 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.595773  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.616719  9322 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:21.597448  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.583043  8582 raft_consensus.cc:2376] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10. Re-sending same reply.
I20260709 13:03:21.580096 13242 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.599503  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.609789  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.615543  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:21.620002  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.616086  8579 raft_consensus.cc:2376] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8. Re-sending same reply.
I20260709 13:03:21.622082  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.622797  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.623253  8581 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.590231 12969 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.595216 13125 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.617738  8583 raft_consensus.cc:2376] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10. Re-sending same reply.
I20260709 13:03:21.624140  8577 raft_consensus.cc:2376] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10. Re-sending same reply.
I20260709 13:03:21.594641 13114 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:21.567063 13235 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 13:03:21.628347  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.527915 13148 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.631596  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.638123  8580 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:21.638172  8577 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:21.633841  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:21.648713  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.652184  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.653080  8575 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:21.665314  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.666019  8580 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:21.666401  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.667095  8582 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:21.621383 12913 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.620925 13236 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:21.685052  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.686151  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.686865  8724 raft_consensus.cc:2376] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11. Re-sending same reply.
I20260709 13:03:21.626363 13063 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:21.690989  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.698987  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.627095 13251 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.700394  8579 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:21.701366  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.706274  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.715626  8577 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:21.718024  8659 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 13:03:21.717865  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.719265  8724 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:21.721657  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.722259  8575 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.626004 13167 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.612413 13137 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.718655  8659 pending_rounds.cc:85] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 4
I20260709 13:03:21.725469  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.729417  8659 pending_rounds.cc:107] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 13:03:21.729648  8579 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:21.729733  8659 raft_consensus.cc:2889] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:21.713451 13063 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:21.731269  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.629910 12969 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.731904  8583 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.734463  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:21.706348 13236 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:21.741770  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.738394  8724 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:21.624776 13238 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.700446 12913 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.684803 13278 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.756676  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.769897  8579 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:21.759086  8655 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:21.759531  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:21.766453  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.774518  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.775344  8577 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.767906  8575 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:21.781193  8580 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
W20260709 13:03:21.752452 13258 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:03:21.787392  8657 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
W20260709 13:03:21.778169 13295 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 13:03:21.807255  9325 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:21.807183  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.809031  8575 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:21.810297  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:21.811214 13303 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 13:03:21.811986  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.813220  8575 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:21.815120  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.817682 13063 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.817888  8575 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:21.819880 13236 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.828750 13278 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.830096 12963 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.731386 13283 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.231s
I20260709 13:03:21.839273  8583 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:21.828625 13183 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.848289  8584 raft_consensus.cc:2368] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:21.849491 13183 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.846967  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.812347 13302 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:21.813336  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.844118  8577 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:21.861320  8732 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.849536 12268 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.862308  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:21.862373 12268 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:21.812747 13300 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 13:03:21.717891 13251 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:21.895226  8576 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:21.896692 13251 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:21.899459 13167 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.894798 13138 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:21.908919  8576 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:21.874673 12963 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.917871 12963 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.871472  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:21.920634  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49253.3us
W20260709 13:03:21.884760 13180 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.663s	user 0.006s	sys 0.001s
I20260709 13:03:21.921280  8625 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:21.922039 13018 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:21.922461 13018 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.922789 13018 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.858477 13286 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:21.888197  8580 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:03:21.923964  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:21.924566  8654 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.925159  8580 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:21.926749  8580 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:03:21.928288  8580 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: [8.8-10.9]
I20260709 13:03:21.928779 13281 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:21.929320 13281 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:21.929484  8580 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: [7.7-8.8]
I20260709 13:03:21.930275 13281 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.931479  8580 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:21.931756  8654 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:21.886770 12969 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.910344  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.911152  8742 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-f4f350f118fc449fb61357528a6b14f7 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:03:21.912436 13302 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.913760  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:21.947258  8726 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:21.948400  8582 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:21.916697 12913 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.920078 12963 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:21.894024 13238 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:21.937057  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 74836.1us
I20260709 13:03:21.936409  8581 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:21.940742  8650 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:21.943954  8651 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:03:21.908740 13147 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:21.945868  8579 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:21.914695  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:21.950371  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:21.966787 12268 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:21.968139  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.982069  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:21.982504  9326 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:21.982992  8579 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:21.962145  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.984743  8697 leader_election.cc:400] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
I20260709 13:03:21.985251  8697 leader_election.cc:403] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:03:21.986261  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.990635  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.991714  9322 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:21.994495  8724 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:21.995661  8732 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:21.995798  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:21.997663  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:21.998380  9329 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:21.989307  8580 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:21.946992 13274 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:22.004628 12268 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.005131 12268 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:21.999716  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:22.013868  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51824.9us
I20260709 13:03:22.015936  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:22.019627  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:22.024748  8733 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.000s
I20260709 13:03:22.025293  8733 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:22.026562  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:22.027407  8580 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.028534 12268 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:22.037734  8724 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:22.039705  8577 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:21.968057  8742 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-f4f350f118fc449fb61357528a6b14f7 on tablet f4f350f118fc449fb61357528a6b14f7 with peer 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:22.013947 13247 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.544s	user 0.008s	sys 0.002s
I20260709 13:03:22.052557  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:22.013321 13238 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:22.053385  8580 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:22.013077 13246 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.043076 13240 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:22.038753 12975 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
I20260709 13:03:22.021636 13255 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:21.969290 13283 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 13:03:22.038586 13278 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:22.056954 13158 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:22.062290 13305 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.210s	user 0.002s	sys 0.002s
I20260709 13:03:22.002136 12963 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 13:03:22.086052  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:22.092896  8580 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:22.054564 13247 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.089131 13238 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:22.088349 12969 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:22.055611 13138 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 13:03:22.102831  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:22.113202  8579 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:22.102936  8656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.009s	sys 0.006s
I20260709 13:03:22.108372  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:22.115813  8656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.009s	sys 0.006s
I20260709 13:03:22.116430  8580 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:22.091616 13018 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.083067 13246 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.128381  8579 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.136394 13238 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.138609 13247 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:22.139878 12969 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:22.126102  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:22.146032  8580 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:22.152483 13311 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 13:03:22.152209 13125 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.155000 13125 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:22.155133 13302 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:22.155635 13280 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.650s	user 0.007s	sys 0.003s
I20260709 13:03:22.117250 13255 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.145679 12969 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:22.159868 12171 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:22.160308  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:22.161453 13245 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.802s	user 0.010s	sys 0.002s
I20260709 13:03:22.162029  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:22.191851  8581 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:22.162101  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:22.170042  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:22.173090  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:22.173193  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:22.179421  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:22.226477  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:22.231660  8653 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:22.232017  8658 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:22.168411  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:22.233220  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64866.8us
I20260709 13:03:22.234216  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:22.190922  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:22.193284 13156 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.009s	sys 0.018s
I20260709 13:03:22.144037 13141 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:22.227089  8650 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:22.230168  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 234448us
I20260709 13:03:22.251277  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:22.253892  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:22.254374  8732 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:03:22.254575  8729 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:22.218601 13282 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.000s	sys 0.015s
W20260709 13:03:22.241245  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 241307us
I20260709 13:03:22.180003 13183 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:22.224745  8579 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
W20260709 13:03:22.241972 13156 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.009s	sys 0.018s
I20260709 13:03:22.262432  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:22.265065  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:22.294919  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:22.272703  8581 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:22.295639  8732 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:22.297183  8580 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:03:22.298097  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.298203 12171 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:22.298583  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.298710 12171 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.299315  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:22.279382  8653 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
W20260709 13:03:22.261842  8696 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.585s (SENT)
W20260709 13:03:22.300441  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:22.257722  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84631.3us
W20260709 13:03:22.290355  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 375690us
I20260709 13:03:22.301488  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:22.301347  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:22.300266  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:22.284091 13282 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.000s	sys 0.016s
I20260709 13:03:22.302232  8651 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:22.318090  8622 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.598s (SENT)
I20260709 13:03:22.259505 13167 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:22.259300 13245 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:22.310433  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:22.322570  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:22.324785  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:22.225217 13255 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.341977 12171 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:22.280985 13248 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:22.331807  8653 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:03:22.358772  8726 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:22.361497  8729 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
W20260709 13:03:22.364166  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53782.6us
W20260709 13:03:22.376230  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 128553us
I20260709 13:03:22.303790 13274 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:22.381639  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:22.383489  8654 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:22.342767 13235 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:22.384477  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:22.384883 13323 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:03:22.385190  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 123371us
I20260709 13:03:22.385388  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:22.385890  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:22.360558 13242 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.731s	user 0.010s	sys 0.001s
I20260709 13:03:22.282434 13141 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.383874  8651 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:22.291841 13302 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260709 13:03:22.342550 13318 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:22.395723  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:22.406759 13147 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:22.386482 12171 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:22.360538 13245 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:22.377626 13283 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:22.394858 13158 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:22.410621 13018 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:22.410722  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92686.6us
I20260709 13:03:22.414496 13326 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:22.415928 13246 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:22.418156 13310 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 13:03:22.418259  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:22.397750 13242 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.449348  8729 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.381426 13248 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.433992  8580 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:03:22.439643  8579 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 13:03:22.426608 13148 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.013s	sys 0.003s
I20260709 13:03:22.441560  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:22.444733  8622 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.624s (SENT)
I20260709 13:03:22.439828 13286 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:22.456700  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:22.456720  8650 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:22.462160  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:22.441790 13245 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.462860  8576 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.463016  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:22.463666  8579 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:22.420434 13328 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:03:22.464246  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 160791us
I20260709 13:03:22.382216 12969 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.476094  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:22.476514  8578 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:22.474033  8726 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:22.482039  8729 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:22.461707 13148 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.013s	sys 0.003s
I20260709 13:03:22.486145 12969 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:22.487291 13324 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:22.456476 13326 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:22.495661  8576 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.497299  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:22.499020 13286 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:22.501631  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 259032us
I20260709 13:03:22.531165  9297 leader_election.cc:400] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
I20260709 13:03:22.532157  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:22.508109  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 112457us
I20260709 13:03:22.495071  8729 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:22.521142  8579 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:22.523792  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:22.535780  8726 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.485085 13323 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:22.551779  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:22.523435 13329 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:22.563349  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:22.565889  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 121179us
I20260709 13:03:22.572281  8726 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.545426 13158 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.576887  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:22.542111 13324 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 13:03:22.589296  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:22.591333  8728 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:22.596755  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:22.598052 12171 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:22.598570 12171 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:22.599020 12171 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:22.568089 13341 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:22.586925 13158 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:22.525687 13274 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.595418 13018 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:22.616290  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:22.619108  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 87431.4us
I20260709 13:03:22.623831 13333 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:22.625890 13156 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:22.590353 13336 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:22.626997 13336 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:22.627380  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:22.608361 13158 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:22.628233 13156 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:22.619292 13326 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:22.630232 13317 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 13:03:22.604951 13313 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 13:03:22.603670 13329 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:22.605389 13341 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:22.634398 13240 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.000s
I20260709 13:03:22.646837 13236 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.654666 13333 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.654551 13236 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.655212 13333 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.649214 13329 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:22.657155 13329 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.649691 13141 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:22.661573 13338 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 13:03:22.657068 13236 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:22.625922 13242 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:22.666409 13335 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 13:03:22.671839 13318 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:22.620756 13255 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:22.652311 13330 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.231s	user 0.000s	sys 0.002s
W20260709 13:03:22.720834  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 90185.9us
W20260709 13:03:22.722263  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.681916  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:22.648900 12171 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.631100 13328 raft_consensus.cc:2764] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
W20260709 13:03:22.686128  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:22.687688  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 139780us
I20260709 13:03:22.730800  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:22.731156 12171 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:22.648003 13323 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.594278 13331 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 13:03:22.653568 13341 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:22.732304  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50489us
I20260709 13:03:22.741740  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:22.733024  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 169900us
I20260709 13:03:22.731794  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:22.686700 13240 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:22.744343  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:22.745384  8731 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:03:22.745833  8731 pending_rounds.cc:85] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:03:22.746045  8731 pending_rounds.cc:107] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 9.9
W20260709 13:03:22.675954  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 128815us
I20260709 13:03:22.737913  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:22.746322  8731 raft_consensus.cc:2889] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:22.738762  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:22.746649  8729 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.739414  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:22.746738  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:22.747447  8732 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.685900 13238 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.750325  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:22.750921  8733 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.751719  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:22.748472  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:22.748318  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:22.722818 13318 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:22.753703 13343 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:22.754397 13246 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.754320 13333 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:22.756251  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:22.757700 13304 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:22.759099  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:22.759865  8654 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:22.617353 13286 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.767253  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:22.772804 12171 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:22.773912  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:22.776284  8729 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.782526  8732 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.738226 13324 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.785452  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2739) took 1184 ms. Trace:
I20260709 13:03:22.740267 13242 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:22.738457 13328 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:22.791584  8726 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:22.791249  8575 rpcz_store.cc:276] 0709 13:03:21.600602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.820377 (+219775us) service_pool.cc:224] Handling call
0709 13:03:21.820743 (+   366us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:21.821179 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:21.821208 (+    29us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:21.825332 (+  4124us) log.cc:844] Serialized 78 byte log entry
0709 13:03:22.179679 (+354347us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:22.179736 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:22.179832 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:22.785044 (+605212us) raft_consensus.cc:1685] finished
0709 13:03:22.785098 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:22.785333 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2226,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":352519,"threads_started":1,"wal-append.queue_time_us":718221}
I20260709 13:03:22.748592 13018 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.748684 13236 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:03:22.795323  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50887.9us
I20260709 13:03:22.796594 12171 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:22.797485 13347 raft_consensus.cc:2764] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 13:03:22.797066 13245 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.762418 13158 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.739833 13148 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 13:03:22.740212 13141 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 13:03:22.794960 13328 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:22.794535 13242 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:22.788152 12969 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.805819  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 876) took 1265 ms. Trace:
I20260709 13:03:22.815281 12171 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:22.815510  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 68847.5us
I20260709 13:03:22.816180  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:22.803807 13334 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.068s	user 0.002s	sys 0.000s
I20260709 13:03:22.819049 13158 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.814071 13018 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.816464 13236 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:22.755043 13246 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.820803  8578 rpcz_store.cc:276] 0709 13:03:21.540673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.542366 (+  1693us) service_pool.cc:224] Handling call
0709 13:03:22.805620 (+1263254us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b54002cb0d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:22.805760 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2145466624}
I20260709 13:03:22.821903 12171 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:22.822391  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:22.823107  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:22.778406 13238 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:22.898321  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 75282us
I20260709 13:03:22.821617 13242 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:23.048062 13352 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.048686 13352 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.826092 13282 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:22.754293 13343 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.049582 13282 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:22.825811 12969 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.903733 13246 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:22.904392 13347 raft_consensus.cc:2764] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 13:03:22.904528 13236 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:22.821563 13328 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:22.798669 13324 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:22.913443 13018 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:22.915151 13326 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:22.918344 13238 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:22.897580 13331 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.983237  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:23.118896  8651 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.924057 13245 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:23.009302  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 186965us
I20260709 13:03:22.983098 13099 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:23.009264  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 235401us
I20260709 13:03:23.120656 13099 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.003968 13339 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:23.007184 13345 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:23.121363  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.123181  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.033982 13348 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:23.039932 13355 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:23.041664 13278 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:23.042657 13286 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.097425 13333 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:23.097726 13343 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.126209 13331 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.136365  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.137243  9328 raft_consensus.cc:2410] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:03:23.139285 12969 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.132100 13278 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:23.118158 13326 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.140210  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.140506 13278 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.111215 13324 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.121968 13345 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.127153 13099 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.154578 13345 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.121702 13339 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.157502 13318 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.157855 13339 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.158950 13333 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.160105 13278 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.160933 13352 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.130604 13246 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:23.162112 13246 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:23.163362 13246 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.149528 13286 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.137136 13331 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:23.165433  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.165953 13333 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:23.116055  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 319038us
I20260709 13:03:23.119843  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.169498 13339 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:03:23.171010  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51235.9us
I20260709 13:03:23.172025  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.170181 13339 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:23.174635 13339 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.177561  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:22.857532 13063 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:23.127534  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:23.178982  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51518.6us
I20260709 13:03:23.179471 13242 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.180805 13099 raft_consensus.cc:2764] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 13:03:23.181718 13099 raft_consensus.cc:2764] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 13:03:23.050575 13282 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.183136 13018 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:23.183689 13018 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:23.131984 13148 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:23.183964  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.184453 13148 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:23.185163 13099 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:23.185374 13148 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.185918 13099 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:23.184693 13018 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.187170 13099 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.133978 13355 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.134435 13158 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.159618 13318 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.183773 13282 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:23.141680 13326 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:23.189755  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.127153 12171 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.190353  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.196585  8696 leader_election.cc:400] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:23.209487  8696 leader_election.cc:403] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:03:23.203142  8651 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:23.195457 13242 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.210152  8658 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:23.210651  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2744) took 1606 ms. Trace:
I20260709 13:03:23.211179  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.211033  8577 rpcz_store.cc:276] 0709 13:03:21.602589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.861634 (+259045us) service_pool.cc:224] Handling call
0709 13:03:21.864388 (+  2754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:22.040267 (+175879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:23.208784 (+1168517us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54000f91d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:23.208945 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":402695424}
W20260709 13:03:23.208058 13329 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.550s	user 0.007s	sys 0.004s
I20260709 13:03:23.216310  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.196751 13063 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:23.227617  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.228281  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.228979  8731 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:23.232385  8730 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:23.212867 13318 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:23.211405 13331 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.231381  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:23.213578 13158 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:23.235005  8623 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:23.215883 12969 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:23.222681 13328 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.224858 13063 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:23.246321  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:23.220227 13352 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 13:03:23.236132 13318 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.255544 13345 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:23.257313 13355 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.262427  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.241282 13158 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.248580 13347 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:23.227080 13329 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.213654 13341 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.272243  8733 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:23.248157 13125 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:23.235323 13245 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20260709 13:03:23.285101  9297 leader_election.cc:336] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.846s (SENT)
I20260709 13:03:23.276407 13158 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.271159 13318 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.241496 13331 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:23.311304  9297 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.917s (SENT)
I20260709 13:03:23.311980  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.252622 13323 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:03:23.315553 13363 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:23.316306 13363 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.247506  9322 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "04399390cf414886a3453981dc1f74aa"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:23.318945  8731 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:23.319527  9322 raft_consensus.cc:606] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
W20260709 13:03:23.318945  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 75961us
I20260709 13:03:23.284271 13347 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:23.339210  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:23.340211  8651 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:23.343948  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 103624us
I20260709 13:03:23.326924 13274 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:03:23.349385  8651 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:23.337041 13286 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:23.337479 13328 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:23.350850 13360 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.123s	user 0.000s	sys 0.002s
I20260709 13:03:23.353638 13147 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:23.372145 13363 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.373730  9322 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:23.374488  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.375180  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:23.385998  9328 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:23.375306  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:23.378032  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 870) took 1468 ms (client timeout 1917 ms). Trace:
I20260709 13:03:23.378324 13331 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.378185 13063 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.414120  8654 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:23.414458 13063 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.414712 13158 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.415326  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:23.415657  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:23.416123  8654 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:23.416355  8578 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:23.417035 13158 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:23.386911  8728 rpcz_store.cc:269] 0709 13:03:21.909638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.910030 (+   392us) service_pool.cc:224] Handling call
0709 13:03:23.377823 (+1467793us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b5400416254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:23.377932 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":645244160}
I20260709 13:03:23.395439  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.395650  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:23.422084  8579 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:23.396595  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.397462  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:23.423525  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46550.2us
I20260709 13:03:23.380848 13347 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.428495  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:23.411597 13341 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.413167  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.411643  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.383212  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:23.429314  8579 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:23.425462  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.432276  8728 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:23.428017  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.410884  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.433992  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:23.434633  8583 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:23.437119  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.437923  8581 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:23.438129  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.438316 13366 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:23.438884  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.439267 13366 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:23.439550  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.439729 13366 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:23.438129  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:23.440891  8577 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:23.442031  8656 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:23.437320  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.444168  8582 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:23.446094  8651 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:23.448675  8579 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:23.449894  8581 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:23.451419  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1065) took 1480 ms. Trace:
I20260709 13:03:23.456769  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.458981  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.461516  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.462198  9331 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:23.465683  8730 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:23.452004  9323 rpcz_store.cc:276] 0709 13:03:21.952939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.954536 (+  1597us) service_pool.cc:224] Handling call
0709 13:03:21.990057 (+ 35521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:22.019244 (+ 29187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:22.019274 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:23.233062 (+1213788us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b2c00047a38. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:23.235600 (+  2538us) log.cc:844] Serialized 26 byte log entry
0709 13:03:23.320973 (+ 85373us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:23.321027 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:23.321800 (+   773us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:23.418087 (+ 96287us) raft_consensus.cc:1685] finished
0709 13:03:23.418109 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:23.433655 (+ 15546us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16732,"raft.queue_time_us":92341,"spinlock_wait_cycles":48515584,"thread_start_us":57929,"threads_started":1,"wal-append.queue_time_us":94886}
I20260709 13:03:23.467546  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:23.468191  8728 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.471131  8575 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:03:23.469624  8578 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:23.482218  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.482928  8578 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:23.484004  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.484949  9329 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:23.487218  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.487571  8577 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:23.450237 13341 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.483243  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.492743  8656 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:23.489141  8583 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:23.455557 13063 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.494653  8582 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:23.456333 12975 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:23.427974 13147 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.508419  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:23.510396  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:23.513873  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.502853  8728 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:23.503962  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:23.516305  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.504563  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.518314  8730 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:23.517549  8654 raft_consensus.cc:2368] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
I20260709 13:03:23.534444  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:23.491560 13333 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.536804  8582 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.466922 13331 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:03:23.539074  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.545178  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.541971  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.523772 13283 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.535619 13018 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:23.538259 13099 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:23.559266  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:23.504801 13063 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.536202 13368 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:23.562839  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.605940  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.534214 13158 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.010s
I20260709 13:03:23.564329  8654 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:23.565140  8730 raft_consensus.cc:2376] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 11. Re-sending same reply.
I20260709 13:03:23.610117  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.520013 12975 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.577493  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:23.614873  9331 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:23.578353  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:23.578791 13278 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:23.580729  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:23.577862  9297 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:23.619916  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.537606 13329 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.573217  8582 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:23.583618 13318 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.564074 13099 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:23.586664 13147 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.564258 13148 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 13:03:23.593063  8659 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:23.544214 13333 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.607897  9328 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:23.611840  8651 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:23.501950 13341 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.622684  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.627511 13180 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:23.605083 13368 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:23.665280  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.608706 13018 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:23.683470  8654 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:23.602970  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:23.686252  8730 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:03:23.613540 13125 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:23.633131 13099 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:23.686980 13278 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.697207  8657 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:23.624215 13282 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:23.626715 13286 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:23.625255 13246 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:23.707981  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 105120us
I20260709 13:03:23.722640  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.710218 13063 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:23.727612  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.711395 13125 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.730518  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:23.732230 13125 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.706003 13368 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:23.713902 13348 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.714923 13245 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:23.744854 13245 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:23.729311 13347 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.723826 13180 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:23.723623 13278 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:23.750176  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 139280us
I20260709 13:03:23.734673 13063 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:23.735249 13339 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:23.733739 13282 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:23.721225 13286 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:23.748553 13375 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 13:03:23.758280 13355 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.010s
I20260709 13:03:23.762099 13286 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.810431  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.743137 13348 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:23.820567  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 197923us
I20260709 13:03:23.772202 13141 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.598834 13283 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.770560 13355 consensus_queue.cc:1243] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260709 13:03:23.825438  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 94987.4us
I20260709 13:03:23.846089  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:23.847397  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:23.752876 13347 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.757615 13180 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.543792 13366 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.825699 13242 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:23.862998  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2340) took 1564 ms. Trace:
I20260709 13:03:23.766665 13018 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.821110 13147 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 13:03:23.863751  8727 rpcz_store.cc:276] 0709 13:03:22.299002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:22.299129 (+   127us) service_pool.cc:224] Handling call
0709 13:03:22.299415 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:22.299623 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:22.343027 (+ 43404us) log.cc:844] Serialized 14 byte log entry
0709 13:03:23.814558 (+1471531us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b18003d0000. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:03:23.862655 (+ 48097us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:23.862709 (+    54us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:23.862760 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:23.862826 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:23.862939 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26281,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":2909,"spinlock_wait_cycles":44845440,"thread_start_us":47768,"threads_started":1}
I20260709 13:03:23.862257 13278 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:23.860404 13318 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.886873 13180 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.864490 13339 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:23.887701  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.888295  8727 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:03:23.888443  8655 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:23.892696  9331 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:23.895704 13387 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:23.896755 13341 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:23.898125 13380 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
W20260709 13:03:23.899652 13379 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:03:23.873144 13283 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.876706 13248 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.846303 13141 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.860154 12975 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.901700  9322 raft_consensus.cc:606] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:23.907106  9322 raft_consensus.cc:614] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:23.880538  8731 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:23.863806 13245 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:23.902949 13246 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:23.917371 13283 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:23.918342 13283 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.919960  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.920745 13283 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.922571  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:23.930312 13141 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:23.931371 13362 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.551s	user 0.003s	sys 0.000s
W20260709 13:03:23.936208  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.945497 13099 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.948190 13099 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.949259  8733 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:23.867705 13333 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:23.956569  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.965497 13183 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
W20260709 13:03:23.968657  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 868) took 2209 ms (client timeout 1846 ms). Trace:
I20260709 13:03:23.968662 13183 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:23.969493 13183 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:23.969442  8724 rpcz_store.cc:269] 0709 13:03:21.758836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.759253 (+   417us) service_pool.cc:224] Handling call
0709 13:03:23.968316 (+2209063us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54000ea454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:23.968522 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":513731968}
I20260709 13:03:23.970753 13183 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:23.971634 13347 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:23.972270 13183 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.972599  8732 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:03:23.973547 13283 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:23.974380 13283 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:23.975703 13283 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.979967  9331 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 13:03:23.980418  9331 pending_rounds.cc:85] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 7
I20260709 13:03:23.980691  9331 pending_rounds.cc:107] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 13:03:23.980924  9331 raft_consensus.cc:2889] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:23.982023  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.982713  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:23.983280  8651 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:23.982082  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:23.984192  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:23.984704  9326 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:23.982262  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:23.985862  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:23.988296  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:23.988803  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 141488us
I20260709 13:03:23.989828  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:23.963099 13147 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.867537 13282 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:23.992405  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:23.990207 13236 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:23.997177  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:23.951967 13245 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.018770  9325 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:23.985004 13137 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:23.994165 13304 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:24.027448  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.028728  9323 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:23.990387 13386 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:23.994340 13099 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:24.024076  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:24.047115  9331 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 7 LEARNER]: Advancing to term 9
I20260709 13:03:24.050413  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.051312  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:24.054196  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 106912us
I20260709 13:03:23.995381 13147 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.056790  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.062403  9323 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:24.036248 12963 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:24.063190  9327 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:24.035501 13099 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:24.039500 13236 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:24.042834 13386 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:24.069737  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.070392  9325 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:24.072496  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.056857 13340 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:24.073246  9323 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:24.075338  8575 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:03:24.072471  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 82621.2us
I20260709 13:03:24.027150 13137 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.077869  9331 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:03:24.058766 13246 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.072151 13236 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.081810 13236 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:24.084614  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:24.085831  8724 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:24.099278  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.100504  8577 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:24.101761  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.067502 13099 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:24.130935 13368 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.162837  8579 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:24.164203  8577 raft_consensus.cc:1217] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Deduplicated request from leader. Original: 3.3->[4.4-13.7]   Dedup: 4.4->[10.5-13.7]
I20260709 13:03:24.164537 13368 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.164721  8577 pending_rounds.cc:85] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 4
I20260709 13:03:24.165019  8577 pending_rounds.cc:107] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 13:03:24.165304  8577 raft_consensus.cc:2889] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:24.123883  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.082963 12963 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:24.071623 13248 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.157176  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.081826 13390 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:24.160666  8576 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:24.057165 13331 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 13:03:24.056916 13138 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 13:03:24.096347 13341 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:24.194126 13352 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:24.194252 13341 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.195750 13368 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:03:24.147914 13388 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 13:03:24.196362  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.167097  8655 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:24.167626  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.199256  8724 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:24.200079  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.200253  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.201004  9331 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 7: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:24.166543 13137 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:24.179395 13391 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:03:24.179157 12963 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.210206  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.181175 13366 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:24.181559  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.180152 13248 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.182428 13397 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:24.185739  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:24.187580  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2801) took 1380 ms. Trace:
I20260709 13:03:24.220824  8658 rpcz_store.cc:276] 0709 13:03:22.806945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:22.807476 (+   531us) service_pool.cc:224] Handling call
0709 13:03:22.817972 (+ 10496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:23.210594 (+392622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:24.187227 (+976633us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:24.187456 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":901451136}
I20260709 13:03:24.192619  8576 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:03:24.103649 13340 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:24.226631  8579 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:24.206727  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 103126us
I20260709 13:03:24.216156  9323 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:24.216760  8725 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:03:24.218727  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.228801  9327 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:24.223235  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.196386 13341 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:24.231052  9322 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:24.231980  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:24.235334  8468 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d0a2b9c938d94a4fa258e068ea054d84 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:24.230170  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:24.239400  8576 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:24.243300 13364 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.872s	user 0.003s	sys 0.000s
I20260709 13:03:24.243238 13137 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:24.243872 13137 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:24.244160 13125 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.503s	user 0.006s	sys 0.004s
I20260709 13:03:24.244266 13137 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:24.244642 13125 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.246169 13340 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.247136 12171 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:24.247944  8726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2347) took 1514 ms (client timeout 1693 ms). Trace:
W20260709 13:03:24.248340  8726 rpcz_store.cc:269] 0709 13:03:22.733389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:22.734223 (+   834us) service_pool.cc:224] Handling call
0709 13:03:24.247794 (+1513571us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x7b54001c9154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:24.247889 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106964864}
I20260709 13:03:24.249392  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.250005  8726 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:24.251932  8726 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:24.253583  8730 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:24.258029  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.257937  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:24.258617  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.259374  9331 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:24.259583  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:24.269196  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.270561  8576 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:24.266769  8548 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:24.273561  8652 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:24.273307  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.276506  9322 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:24.278419  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:24.282042  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2798) took 1746 ms. Trace:
I20260709 13:03:24.269033 13341 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.288218  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.283269  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:24.288866  8658 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:24.280467 13242 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:24.291127 13242 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:24.291654 13242 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:24.282476  8650 rpcz_store.cc:276] 0709 13:03:22.535653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:22.535812 (+   159us) service_pool.cc:224] Handling call
0709 13:03:22.545944 (+ 10132us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:22.551021 (+  5077us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:22.551053 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:22.567173 (+ 16120us) log.cc:844] Serialized 26 byte log entry
0709 13:03:23.894190 (+1327017us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b18000298c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:23.990730 (+ 96540us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:23.990784 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:23.990877 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:24.276858 (+285981us) raft_consensus.cc:1685] finished
0709 13:03:24.276884 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:24.281982 (+  5098us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3740,"raft.queue_time_us":2363,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":11393,"spinlock_wait_cycles":2812160,"thread_start_us":96286,"threads_started":1,"wal-append.queue_time_us":1477611}
I20260709 13:03:24.198261 13390 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.295140 13390 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:24.265789 13323 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:24.295818 13323 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.297996 13323 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:24.269274 13345 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:24.299055 13398 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:24.299880 13398 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:24.301962 13398 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:24.302353 13398 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:24.302709 13398 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:24.189272 13386 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.306689  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:24.314127  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 217017us
I20260709 13:03:24.285897 13246 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:24.277383 13405 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 13:03:24.324424  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.288008 13387 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:24.310235 13341 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.331310 13341 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.366393  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.336639 13387 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.327685 13246 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:24.366483 13417 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:24.373682 13406 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 13:03:24.374609 13248 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:24.376670 13377 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.639s	user 0.000s	sys 0.002s
I20260709 13:03:24.377076  9323 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:03:24.327785 13147 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:24.346285 13247 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:24.377743  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.378814  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.384301  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:24.422149  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:24.422796  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:24.382732  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.423329  8578 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:24.400738  9327 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:24.422755 12171 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:03:24.415585 13286 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.596s	user 0.007s	sys 0.006s
I20260709 13:03:24.414224 13280 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.443938  9327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:24.451956  9327 raft_consensus.cc:606] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:24.452340  9327 raft_consensus.cc:614] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:24.413832 13331 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 13:03:24.428333 13282 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:03:24.428555 13141 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:24.453857 13137 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.429221 13220 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 13:03:24.443539 13286 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.466342  9326 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:24.466418  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:03:24.467056 13286 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:24.471854 13387 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.472463 13282 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:03:24.454201 13125 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.473162 13242 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.472352  8578 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:24.479925  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.483400  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.484660  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.486730  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2155) took 1034 ms. Trace:
I20260709 13:03:24.487186  8581 rpcz_store.cc:276] 0709 13:03:23.452543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.452783 (+   240us) service_pool.cc:224] Handling call
0709 13:03:23.461853 (+  9070us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:23.471812 (+  9959us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:23.471843 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:23.481755 (+  9912us) log.cc:844] Serialized 26 byte log entry
0709 13:03:23.560444 (+ 78689us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:23.700164 (+139720us) log.cc:844] Serialized 14 byte log entry
0709 13:03:23.711453 (+ 11289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:23.711554 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:24.376009 (+664455us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:24.486432 (+110423us) raft_consensus.cc:1685] finished
0709 13:03:24.486538 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:24.486673 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11020,"prepare.queue_time_us":213577,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":2791,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":6431,"spinlock_wait_cycles":4245120,"thread_start_us":154546,"threads_started":2,"wal-append.queue_time_us":175648}
W20260709 13:03:24.488840  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:24.489653  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:24.426474 13247 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.492245 13247 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:24.492992 13336 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:03:24.494308 13419 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:03:24.474332 13280 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.494614  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:24.495425  8731 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:24.496384 13280 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.496502  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:24.498225  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:24.500787 13286 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:24.410892 13417 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.509739  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:24.509793 13413 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:03:24.512143 13398 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.465246 13137 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.516578  8728 raft_consensus.cc:2410] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 13:03:24.516366  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.501248 13286 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:24.505818 13387 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.527194 13398 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.523919 13336 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:24.543969  8731 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:03:24.517426  8581 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:24.520882 13286 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:24.560788  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:24.562016  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:24.562628  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:24.564776  8581 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:24.542435 13255 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.596213  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:24.502557 13246 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.568755 13255 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.607856 13255 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.553076 13387 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.616719 13387 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:24.629427  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:24.631186  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:24.631843  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 203963us
I20260709 13:03:24.632062  8584 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:24.637665  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:24.638290  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 259961us
I20260709 13:03:24.638880  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.645314  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.647140  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2620) took 1070 ms. Trace:
I20260709 13:03:24.651371  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.653275  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.655519  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.638994  8584 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:24.588677 13099 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.647699  8733 rpcz_store.cc:276] 0709 13:03:23.576255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.580380 (+  4125us) service_pool.cc:224] Handling call
0709 13:03:24.646911 (+1066531us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b54004722d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:24.647065 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":759393152}
I20260709 13:03:24.667090  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.670095  8584 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:24.670150  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.633752 13331 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:24.691004  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.692708  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.672940  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.671731  8584 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:24.694739  8650 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:03:24.695715  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:24.697985  8581 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:24.697867  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 127150us
I20260709 13:03:24.664440 13125 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.717267 13125 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:24.713510  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:24.720402  8658 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 13:03:24.708685 13378 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.956s	user 0.001s	sys 0.004s
I20260709 13:03:24.751123  8584 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:24.665722 13280 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:03:24.753190  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:24.793118  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:24.794100  8581 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:24.723752 13125 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:24.720330 13387 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:24.800859  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47748.6us
I20260709 13:03:24.802291 13331 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.802855 13331 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.804105  8698 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:24.804306 13390 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:24.801038 12963 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:24.805003 13331 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:24.806345 13018 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:24.806866 13341 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:24.807248 13390 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:24.807672 12963 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:24.800379  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:24.813107 13424 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
I20260709 13:03:24.814045 13417 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:24.819012 13418 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.349s	user 0.003s	sys 0.000s
I20260709 13:03:24.927549  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:24.823962 13336 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:24.723851 13255 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.101s
I20260709 13:03:24.751714 13368 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:24.841058 13286 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.853180 13348 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:24.809213 13245 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:24.887454  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:24.888475  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 765) took 1177 ms. Trace:
I20260709 13:03:24.890232  8584 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:24.799101 13247 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:24.898721  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2614) took 1404 ms. Trace:
W20260709 13:03:24.900141  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 103239us
I20260709 13:03:24.942560  8654 rpcz_store.cc:276] 0709 13:03:23.711132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.711727 (+   595us) service_pool.cc:224] Handling call
0709 13:03:24.849082 (+1137355us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b540005a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:24.888404 (+ 39322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1570610560}
I20260709 13:03:24.943173  8728 rpcz_store.cc:276] 0709 13:03:23.494797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.565740 (+ 70943us) service_pool.cc:224] Handling call
0709 13:03:24.847592 (+1281852us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b5400089dd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:24.898640 (+ 51048us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2070501632}
I20260709 13:03:24.943523  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:24.886626  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:24.909991  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:24.945464  8581 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:24.946697  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.947314  9330 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:24.909597  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:24.913739  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 113432us
I20260709 13:03:24.948045  8576 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:24.948902  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.949517  8576 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:24.950270  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.910254 12969 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:24.919734 13352 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:24.950364  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:24.909899 13390 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:24.953711  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:24.955929  9328 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:24.962666  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:24.972678 13415 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.503s	user 0.000s	sys 0.002s
I20260709 13:03:24.972193 12969 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:24.978420 12969 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.979241  9325 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:03:24.932054 13331 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:24.967770  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:24.906889 13387 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.002547  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.956629 13341 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:25.002581  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:24.959451 13158 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:24.956933 13336 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.999917 13183 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.009859  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:24.934289 13340 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 13:03:25.007704  8584 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:25.002017 13405 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:25.012308  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44629.7us
I20260709 13:03:25.012951  8581 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:24.960098 13352 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:25.046955  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:25.011519 13341 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.053372  8581 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:25.009418  9328 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:24.980095 13286 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.038573  8577 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 13:03:25.039218  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:24.959774 13099 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:24.960230 13283 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 13:03:25.035928  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:25.005364 13236 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 13:03:25.063781  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:25.036844 13247 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:25.068240  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:25.045203 13352 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:25.044595 13405 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:25.041378 13158 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:25.089972  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 762) took 1654 ms. Trace:
I20260709 13:03:25.005581 13399 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 13:03:25.008284 13331 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.051924 13125 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:25.056017 13341 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:25.080919 13138 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:25.050838 13183 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.090014 13405 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:25.067440 12975 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:25.039160 13336 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.081053 13236 consensus_queue.cc:1243] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:03:25.090442  8652 rpcz_store.cc:276] 0709 13:03:23.435570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.695089 (+259519us) service_pool.cc:224] Handling call
0709 13:03:23.714487 (+ 19398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:24.274010 (+559523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.089638 (+815628us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b5400420c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:25.089881 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1285152512}
W20260709 13:03:25.076068  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 148599us
I20260709 13:03:25.041667 12963 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.051951 13390 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:25.096238  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49393.4us
I20260709 13:03:25.115806  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.102310 13283 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:25.101378 13138 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.127909 13138 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.129798 13158 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.134724 13331 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.137575 13340 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.138149 13340 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:25.139531 13436 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:03:25.139845 12171 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:25.140269 13340 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:25.140365 12171 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.140828 13340 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.141783  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:25.142668 13336 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.143693 13387 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:25.120895  9323 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:25.148275  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:25.125072  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.152329  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:25.153340  8654 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:25.155417  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:25.156539  8654 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:25.141292 13340 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.161885  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:25.163132  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:25.169807  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:25.171379  9296 leader_election.cc:400] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
I20260709 13:03:25.171768  9296 leader_election.cc:403] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:03:25.174971  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.175979  8724 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:25.175933  8581 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:03:25.162629  8654 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:03:25.180678  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.172919  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.181859  8728 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:03:25.182902  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.186928  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.187755  8728 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:25.188719  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:25.189452  8655 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:25.187825  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.194079  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:25.194065  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 78372.4us
W20260709 13:03:25.197187  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:25.198105  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.199705  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.202216  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.202908  8729 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:25.204460  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.198041  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.204493  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.209365  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.209558  8725 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:25.211889  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.212740  8578 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:25.213934  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.222137 12969 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:25.224182 12969 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:25.225476  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.227058  8731 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:25.110783 13445 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 13:03:25.229283  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.110617 12975 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.233711  8656 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:25.230687  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:25.237498  8732 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:25.244813  8729 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:25.162154 13390 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:25.248543  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 68774.9us
I20260709 13:03:25.252533  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.252835  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.253552  8732 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:25.164414 13255 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.253789  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.256238  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.203727 13331 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:25.191690 13387 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.204174 13336 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10.
I20260709 13:03:25.269215  8578 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:25.266531  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:25.203924 13138 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:25.271416  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.204365 13302 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.228660 13245 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:25.282327  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:25.266636 13391 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:03:25.285434  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.288367  9330 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:25.265723 13400 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:25.276297 13416 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:03:25.235641 13283 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:25.273865 13248 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 13:03:25.225150 13405 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.265573 12969 raft_consensus.cc:2764] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 13:03:25.281011 13183 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:25.276528 13331 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.285039  8653 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Committing config change with OpId 7.7: config changed from index -1 to 7, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:03:25.287645  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.268841 13348 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:03:25.315856  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:25.319823  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 125858us
I20260709 13:03:25.276014 13426 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:25.265422 13399 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.040s
W20260709 13:03:25.312687  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69947us
I20260709 13:03:25.297389 13431 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:25.295565 13383 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:25.332532  8581 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:25.305009 13340 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:25.266824 13439 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:03:25.333036  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.275974 13255 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.278446 13282 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 13:03:25.357656  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2165) took 1159 ms. Trace:
I20260709 13:03:25.265666 13158 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 13:03:25.357157  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:25.361847  8581 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:25.329669 13331 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:25.286180 13387 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.369721  8579 rpcz_store.cc:276] 0709 13:03:24.198582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.238257 (+ 39675us) service_pool.cc:224] Handling call
0709 13:03:24.238580 (+   323us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:24.239752 (+  1172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:24.239785 (+    33us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:24.253372 (+ 13587us) log.cc:844] Serialized 78 byte log entry
0709 13:03:24.301267 (+ 47895us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:24.319588 (+ 18321us) log.cc:844] Serialized 14 byte log entry
0709 13:03:24.341466 (+ 21878us) log.cc:844] Serialized 14 byte log entry
0709 13:03:24.379202 (+ 37736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:24.379272 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:24.851979 (+472707us) spinlock_profiling.cc:243] Waited 6.29 ms on lock 0x7b1800448cc0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:25.135640 (+283661us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b1800448cc0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:25.357314 (+221674us) raft_consensus.cc:1685] finished
0709 13:03:25.357345 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:25.357576 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72321,"prepare.queue_time_us":70365,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":73,"raft.queue_time_us":109928,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":16713,"spinlock_wait_cycles":21666176,"thread_start_us":18961,"threads_started":1,"wal-append.queue_time_us":51558}
I20260709 13:03:25.366302  8584 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:25.373961  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:25.333566 13236 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:25.358182 13428 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.298780 13302 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.318150 13018 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
W20260709 13:03:25.387740  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 121565us
I20260709 13:03:25.389374  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.368929 13445 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.392843  8724 raft_consensus.cc:2376] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11. Re-sending same reply.
W20260709 13:03:25.397496  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 113297us
I20260709 13:03:25.396365  8728 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 13:03:25.394343  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 123026us
I20260709 13:03:25.398236  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:25.411940 13456 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
W20260709 13:03:25.412194 13453 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.179s	user 0.001s	sys 0.001s
I20260709 13:03:25.392211 13326 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:25.311632 13400 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.387115 13138 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:25.374151 13245 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:25.444361  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 190626us
I20260709 13:03:25.423916 13431 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:25.457741  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 59595us
I20260709 13:03:25.454806  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:25.458307  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2621) took 1366 ms. Trace:
I20260709 13:03:25.458525  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:25.462071  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:25.444586 13420 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:03:25.420441 12963 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 13:03:25.481791  8730 rpcz_store.cc:276] 0709 13:03:24.091789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.197763 (+105974us) service_pool.cc:224] Handling call
0709 13:03:24.249615 (+ 51852us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:24.254120 (+  4505us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.458059 (+1203939us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b54001c9150. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:25.458234 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96997632}
I20260709 13:03:25.423115 13445 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.508440 13445 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:25.437163 13383 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.510033 13383 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.512192 13325 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:25.512457 13383 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:25.452147 13099 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.513221 13099 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.513948 13470 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:25.514830 13470 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:25.515156 13470 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:25.515270 12975 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:25.515363 13099 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:25.437415 13248 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:25.515985 13470 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.396380 13336 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 13:03:25.517823 13336 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:25.518158 13336 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:25.518420 13302 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:25.457037 13326 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:25.392819 13428 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:25.470340 13441 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
I20260709 13:03:25.470386 13138 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.492215 13242 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:25.492543 13454 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 13:03:25.444391 13465 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:03:25.480726 13467 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:25.494515 13431 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.495661 13282 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:25.497989 13405 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.499917 13400 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:25.500020 13404 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:25.439821 13283 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:25.506140 13458 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:03:25.519757  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.484869  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:25.554582 13138 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:25.491823 13463 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:03:25.518944 13336 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.520591 13325 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:25.533995 13386 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:25.553367 13326 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:25.535511  8725 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:03:25.574921  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 90195.5us
I20260709 13:03:25.592103  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:25.575374  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 55702.3us
I20260709 13:03:25.588770 13400 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:25.627957  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 238660us
I20260709 13:03:25.629473  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:25.567665  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 109213us
I20260709 13:03:25.569186 13404 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:25.556324 13478 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.610800 13474 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:25.631197  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:25.517236 13470 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:25.622514  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:25.574882 13247 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:25.572767 13468 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:25.559412 13341 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.644989  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:25.649770  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:25.570978 13282 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.655082  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.659375  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:25.660287 13400 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:25.660813 13400 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:25.661569  8725 pending_rounds.cc:85] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 3
I20260709 13:03:25.661895  8725 pending_rounds.cc:107] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 13:03:25.662416  8725 raft_consensus.cc:2889] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:25.661811 13400 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.665999  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:25.668934  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.671969 13417 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:25.673589 13183 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.674293 13183 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.672758 13417 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.679930 13417 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.677323 13183 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:25.681258  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.652606 13236 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.685520 13439 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.686218 13439 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:25.688437  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:25.689114  9323 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:25.691632  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:25.693642  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.694736  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:25.695113  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2574) took 1713 ms. Trace:
I20260709 13:03:25.695462  9331 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:25.686069 13236 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.642047 13336 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:25.701292  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.648896 13242 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:25.704443  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.695744  8659 rpcz_store.cc:276] 0709 13:03:23.982228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.984028 (+  1800us) service_pool.cc:224] Handling call
0709 13:03:23.994206 (+ 10178us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:23.994438 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:23.994466 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:24.845156 (+850690us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b2c000b1fc8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:24.918860 (+ 73704us) log.cc:844] Serialized 26 byte log entry
0709 13:03:25.067923 (+149063us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:25.067957 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.068018 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:25.658053 (+590035us) raft_consensus.cc:1685] finished
0709 13:03:25.658072 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:25.695019 (+ 36947us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4989312,"thread_start_us":148606,"threads_started":1,"wal-append.queue_time_us":187007}
I20260709 13:03:25.706629  8727 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
W20260709 13:03:25.706948  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 463759us
I20260709 13:03:25.651163 13468 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.686873 12975 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:03:25.711880  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:25.712518  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:25.641017 13465 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 13:03:25.721290  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:25.721997  9331 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:25.722352  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:25.724511  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:25.690910 13426 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:25.727757  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.699713 13387 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:25.731168  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:25.697512 13472 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:25.732179  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.705528 13242 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.705086 13467 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:25.653286 13445 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 13:03:25.735554  8576 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:25.739511  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.739801  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.740140  8576 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:25.732856  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.741771  8581 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:25.742936  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.743556  8575 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:25.685662 13383 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 13:03:25.736590  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:25.745852  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.737762  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:25.744781  8576 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:25.721552 12975 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.747640  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:25.727102 13462 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:25.650233 13478 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.688831 13474 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260709 13:03:25.748818  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:25.723521 13468 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:25.725194 13245 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:25.724992 13397 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:25.729097 13426 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:25.753417  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 123164us
I20260709 13:03:25.742583 13242 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:25.754206  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:25.772794  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:25.780112  8729 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:25.782001  8729 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:25.775426  8582 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:25.749099 13473 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:25.728650 13247 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 13:03:25.750978 12975 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:25.786788  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:25.749593 13386 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:25.737094 12963 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 13:03:25.768828 13417 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.879184  9328 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:25.865262  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:25.749141 13471 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:25.750622 13156 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:25.892697 13476 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:25.894299 13483 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:25.898087 13387 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:25.914386 13439 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.916219 13429 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:25.920104 13368 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:25.698350 13325 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:25.969390  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 213135us
W20260709 13:03:25.995890  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 223956us
I20260709 13:03:25.923013 13341 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:25.999817  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:26.025677  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:25.998607 13386 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.003963  9325 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:26.006579  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:25.865340 13472 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:26.013156  9328 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:03:25.856980 13428 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:25.776414 13426 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:25.788321 13397 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:25.726694 13404 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:26.000603  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 262980us
I20260709 13:03:26.040350  8583 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:26.069680  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44070.2us
I20260709 13:03:26.028671 13325 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:25.998822 13141 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.871610 13451 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.408s
W20260709 13:03:26.079169  8583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 886) took 1739 ms (client timeout 2032 ms). Trace:
I20260709 13:03:26.016776 13483 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.017163 13417 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.087368 13417 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:26.081883  8583 rpcz_store.cc:269] 0709 13:03:24.339513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.822224 (+482711us) service_pool.cc:224] Handling call
0709 13:03:26.006412 (+1184188us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:03:26.079103 (+ 72691us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159528320}
W20260709 13:03:26.087924 13331 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.691s	user 0.009s	sys 0.000s
I20260709 13:03:26.088090 13255 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.088713 13255 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:26.088479 13331 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.070513 13304 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:26.089839 13304 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:26.090029  9330 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:26.092811 13141 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.095122 13141 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.096081 13236 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.070751 13467 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:26.096848  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:26.097162 13467 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.097584 13467 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:26.098054 13484 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.368s	user 0.000s	sys 0.001s
I20260709 13:03:26.097851 13492 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:26.099445 13492 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:26.103009 13446 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:26.089198 13255 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.105795 12171 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.107445 13255 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:26.095355 13438 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.003s	user 0.004s	sys 0.009s
I20260709 13:03:26.096848  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1077) took 1163 ms. Trace:
I20260709 13:03:26.100860 13490 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:26.115082  9326 rpcz_store.cc:276] 0709 13:03:24.937595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.951724 (+ 14129us) service_pool.cc:224] Handling call
0709 13:03:24.952035 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:25.103817 (+151782us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:25.113667 (+  9850us) log.cc:844] Serialized 14 byte log entry
0709 13:03:25.317794 (+204127us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:25.317849 (+    55us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:25.317895 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.096615 (+778720us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54008b4054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:26.096659 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:26.096783 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10074,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":3051,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":28144,"spinlock_wait_cycles":348387584,"thread_start_us":202987,"threads_started":1,"wal-append.queue_time_us":279866}
I20260709 13:03:26.124337  9323 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:26.025517 13439 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:26.039184 13386 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.122422  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:26.024580 13489 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:26.010994 13476 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.030905 13429 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:26.033039 13431 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.129587  9328 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:26.023468 13399 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.093s
I20260709 13:03:26.026867 13387 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.125530 13404 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:26.111137 13483 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:26.115744 13472 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.128557 13286 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:26.186358 13472 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.150264 13465 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.187069 13465 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.190346  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:26.192289 13368 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:26.192121 13467 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.192791  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.193339  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.193742  8655 raft_consensus.cc:2410] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 13:03:26.193697  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.194275  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.194509  8652 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:26.194911  8654 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:26.175840 13489 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:26.150962 13304 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.147555 13476 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.202100  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.205456  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.205940  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 902) took 1069 ms. Trace:
I20260709 13:03:26.206271  8656 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:26.206375  8577 rpcz_store.cc:276] 0709 13:03:25.136581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.255300 (+118719us) service_pool.cc:224] Handling call
0709 13:03:26.201948 (+946648us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:03:26.205879 (+  3931us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5408512}
I20260709 13:03:26.208453  8650 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:26.209057  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1121) took 1565 ms. Trace:
I20260709 13:03:26.211890  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.212730  8656 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:26.209592  9327 rpcz_store.cc:276] 0709 13:03:24.643230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.643346 (+   116us) service_pool.cc:224] Handling call
0709 13:03:24.690399 (+ 47053us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:24.698158 (+  7759us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:24.698193 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:24.698698 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 13:03:25.053768 (+355070us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:25.053826 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.053922 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:26.094605 (+1040683us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:26.202711 (+108106us) raft_consensus.cc:1685] finished
0709 13:03:26.202738 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:26.208982 (+  6244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2473,"raft.queue_time_us":2672,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5106,"spinlock_wait_cycles":3948416,"thread_start_us":258443,"threads_started":1,"wal-append.queue_time_us":347991}
W20260709 13:03:26.217831  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 133092us
I20260709 13:03:26.167407 13282 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:26.232646  8656 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 13:03:26.151435 13390 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.881s	user 0.013s	sys 0.000s
I20260709 13:03:26.221069  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:26.220026  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:26.185700 13138 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.595s	user 0.005s	sys 0.005s
W20260709 13:03:26.222754  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 197832us
I20260709 13:03:26.198791 13304 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.145813 13439 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.242063  8726 raft_consensus.cc:2410] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 13:03:26.235044 13481 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 13:03:26.243952 13477 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.771s	user 0.007s	sys 0.011s
I20260709 13:03:26.192836 13368 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.259696  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:26.230932 13473 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:26.272498 13399 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.273157 13399 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.275713 13399 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:26.277634  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.277485  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.279002  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.279639  8582 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:26.288278  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:26.288390  8577 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:26.289778 13431 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:26.289059  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:26.292469  8583 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:26.292774  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:26.293977 13431 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:26.294380 13431 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:26.294198  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:26.294843 13431 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:26.295439 13488 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:26.295850 13488 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.296185 13488 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.309139 13331 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.309958 13331 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:26.310326 13472 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:26.311739 13352 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.219s	user 0.011s	sys 0.001s
I20260709 13:03:26.312047 13331 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:26.253558 13496 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 13:03:26.312887 13331 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:26.318056 13467 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.256166 13141 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 13:03:26.318351  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2381) took 1082 ms. Trace:
I20260709 13:03:26.319675 13125 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 13:03:26.332216  8733 rpcz_store.cc:276] 0709 13:03:25.235692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.235820 (+   128us) service_pool.cc:224] Handling call
0709 13:03:25.236133 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:25.389511 (+153378us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:25.437301 (+ 47790us) log.cc:844] Serialized 14 byte log entry
0709 13:03:25.707955 (+270654us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:25.724199 (+ 16244us) log.cc:844] Serialized 26 byte log entry
0709 13:03:25.724377 (+   178us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:25.724426 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.724516 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:26.318072 (+593556us) raft_consensus.cc:1685] finished
0709 13:03:26.318113 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:26.318278 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13073,"prepare.queue_time_us":33921,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":6121,"raft.queue_time_us":67971,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":50895,"thread_start_us":355622,"threads_started":2,"wal-append.queue_time_us":552825}
I20260709 13:03:26.254709 13341 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:26.249027 13282 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:26.312287 13352 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.318414 13485 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.283s
I20260709 13:03:26.344568  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:26.347571  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 274901us
I20260709 13:03:26.312458 13472 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.248060 13138 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.196849 13489 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.320609 13156 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:26.396824 13156 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.397260 13156 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:26.400007  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:26.400636  8576 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:26.359387  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:26.402272  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:26.405153  8733 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:03:26.432783  9322 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:26.352658  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 252248us
I20260709 13:03:26.361218  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.197721 13429 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.374454 13471 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:26.375099 13498 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:03:26.380960 13487 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
W20260709 13:03:26.383663 13486 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 13:03:26.385869 13352 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:26.385948 13397 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.349242 13428 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.386907  8650 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:26.428953 13483 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:26.388916 13475 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.816s	user 0.000s	sys 0.006s
I20260709 13:03:26.390246 13439 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.394083 13387 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:26.395789 13242 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.538s	user 0.007s	sys 0.000s
I20260709 13:03:26.476143 13242 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.477181 13483 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.478420 13242 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:26.394018  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:26.479164  8582 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.479281  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2370) took 1292 ms. Trace:
I20260709 13:03:26.479651  8726 rpcz_store.cc:276] 0709 13:03:25.186532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.186680 (+   148us) service_pool.cc:224] Handling call
0709 13:03:26.479076 (+1292396us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400436554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:26.479224 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2424473472}
I20260709 13:03:26.480857  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:26.408326 13336 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.703s	user 0.008s	sys 0.000s
I20260709 13:03:26.452198 13498 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:26.483048 13498 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.410444 13473 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.486371 13488 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.412770 13489 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:26.413297 13477 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:03:26.416749 13465 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.420627 13386 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.501809  8654 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:03:26.502844 13386 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:26.422889 13468 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.635s	user 0.009s	sys 0.000s
I20260709 13:03:26.503644 13468 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.505977 13468 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.346974 13282 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:26.506618 13282 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:26.425971 13326 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.791s	user 0.010s	sys 0.001s
I20260709 13:03:26.429234 13138 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.360816  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:26.511950  8582 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:26.334837 13331 consensus_queue.cc:1243] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:26.430511 13368 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.435249  8576 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:26.437949  9328 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:26.441035  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.451937  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.463605  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.464871  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:26.377238 12975 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.465931  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:26.474650 13428 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.488852  8726 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:26.452404 13471 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:26.456668 13383 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.492375 13236 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.532207 13236 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.511230  8584 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:26.509666 13326 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.534629 12975 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:26.535609 13326 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.536484  9331 raft_consensus.cc:2410] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 13:03:26.536574  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2631) took 1338 ms. Trace:
I20260709 13:03:26.536991  8727 rpcz_store.cc:276] 0709 13:03:25.198072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.201421 (+  3349us) service_pool.cc:224] Handling call
0709 13:03:25.201734 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:25.708877 (+507143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.536301 (+827424us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b54000e9f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:26.536514 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1160374528}
I20260709 13:03:26.538571  8727 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:03:26.513509  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:26.541357 13503 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 13:03:26.513681  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:26.528475  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 62745.8us
I20260709 13:03:26.545683  8726 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:26.546286  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:26.547678  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:26.548498  8732 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 13:03:26.551417  8732 pending_rounds.cc:85] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:03:26.551743  8732 pending_rounds.cc:107] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 13:03:26.552100  8732 raft_consensus.cc:2889] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:26.553094 13366 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:26.553094  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.553872 13366 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:26.554175 13341 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:26.554781 13366 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:26.554301  8580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 895) took 2068 ms (client timeout 1776 ms). Trace:
I20260709 13:03:26.491942 13476 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:26.559448  8580 rpcz_store.cc:269] 0709 13:03:24.486015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.823976 (+337961us) service_pool.cc:224] Handling call
0709 13:03:26.552917 (+1728941us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:03:26.554223 (+  1306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155345280}
I20260709 13:03:26.579298  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:26.580655  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.581825 13397 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:26.582472 13417 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:26.582365 13397 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:26.583336  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 363347us
I20260709 13:03:26.583144 13417 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.592442  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:26.587513  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:26.597731  8651 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:26.584313  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:26.585388  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:26.569180  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 318821us
I20260709 13:03:26.556627 12969 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:26.554809 13341 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:26.615286  8578 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 11 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:26.622368  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:26.623050  8652 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.621272  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:26.624859  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:26.628815  9331 raft_consensus.cc:2410] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 11 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:03:26.625422  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:26.625381  9294 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.776s (SENT) [suppressed 13 similar messages]
W20260709 13:03:26.637111 13515 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 13:03:26.577937 13368 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:03:26.637778  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:26.637971 13508 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:03:26.645839 13404 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:26.646514 13404 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:26.647318  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:26.648023  8655 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:26.648659 13404 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.638058 13417 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:26.649350  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2823) took 1484 ms. Trace:
I20260709 13:03:26.629647 12969 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:26.639858  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.650894 12969 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:26.614441 13326 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:26.653988 13326 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.656389 13326 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.618973 13325 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:26.657217 13325 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.657635 13325 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:26.602628 13483 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 13:03:26.641912 13156 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.656700 13476 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:26.666846  8655 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:26.617065 13428 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.667439  9330 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:26.667659  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:26.693670  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:26.696273  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 75487.4us
I20260709 13:03:26.698130  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:26.698889  8580 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.670078  8652 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:26.611996 13474 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:26.649793  8653 rpcz_store.cc:276] 0709 13:03:25.164539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.164675 (+   136us) service_pool.cc:224] Handling call
0709 13:03:25.165010 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:25.230329 (+ 65319us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:25.372674 (+142345us) log.cc:844] Serialized 14 byte log entry
0709 13:03:25.478210 (+105536us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:25.489764 (+ 11554us) log.cc:844] Serialized 26 byte log entry
0709 13:03:25.489901 (+   137us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:25.555789 (+ 65888us) log.cc:844] Serialized 14 byte log entry
0709 13:03:25.559198 (+  3409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.559314 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:26.025225 (+465911us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:26.649157 (+623932us) raft_consensus.cc:1685] finished
0709 13:03:26.649181 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:26.649299 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49164,"prepare.queue_time_us":84351,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":65955,"raft.run_cpu_time_us":388,"raft.run_wall_time_us":8473,"spinlock_wait_cycles":4685184,"thread_start_us":76250,"threads_started":1,"wal-append.queue_time_us":76766}
I20260709 13:03:26.611182 13488 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.549916 13383 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:26.664121 13470 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.013s	user 0.012s	sys 0.000s
I20260709 13:03:26.701412  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:26.614713 13426 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.704738  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2808) took 2054 ms. Trace:
I20260709 13:03:26.739754  8658 rpcz_store.cc:276] 0709 13:03:24.650661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.650775 (+   114us) service_pool.cc:224] Handling call
0709 13:03:24.690052 (+ 39277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:24.723283 (+ 33231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.704432 (+1981149us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b54001d1fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:26.704657 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68956416}
I20260709 13:03:26.704785  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:26.616542 13246 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 13:03:26.707464  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:26.708407  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.747414  8728 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:26.752117  9324 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:26.759002  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.776072  8580 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:03:26.789206  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2380) took 1581 ms (client timeout 1809 ms). Trace:
W20260709 13:03:26.790783  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 87463.1us
W20260709 13:03:26.839933  8724 rpcz_store.cc:269] 0709 13:03:25.207335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.225164 (+ 17829us) service_pool.cc:224] Handling call
0709 13:03:26.788942 (+1563778us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b5400339354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:26.789124 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":383311232}
I20260709 13:03:26.794126  8731 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:26.841774  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:26.773099  8651 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:26.830349  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:26.832592  8577 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:26.845177  8652 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:26.624301 13495 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.129s
I20260709 13:03:26.737285 13125 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:26.744164 13478 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:26.758196 13018 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:03:26.845371 13506 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 13:03:26.788208 13283 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:26.787804 13474 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:26.830192 13485 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:03:26.871627  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 164229us
I20260709 13:03:26.872807  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:26.875398  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 250009us
W20260709 13:03:26.875536  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 253428us
I20260709 13:03:26.863716 13491 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:26.672248 13471 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:26.863793 13018 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:26.978210  8655 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:26.881827 13485 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:26.980399  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:26.981722 13520 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 13:03:26.981962 13248 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:03:26.982502 13507 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
W20260709 13:03:26.983395 13509 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.311s	user 0.001s	sys 0.000s
W20260709 13:03:26.983330  8684 heartbeater.cc:646] Failed to heartbeat to 127.7.78.126:44983 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.7.78.126:44983 timed out after 15.000s (SENT)
I20260709 13:03:26.985733  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:27.006493  8658 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 13
W20260709 13:03:26.989590 13431 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.694s	user 0.010s	sys 0.004s
I20260709 13:03:26.992621  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2787) took 1265 ms. Trace:
I20260709 13:03:26.994755  8548 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:26.921705 13495 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:27.002774  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:26.863874 13125 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.019071  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:27.020117  9326 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:27.021842  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:27.023721 13497 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:27.009599  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:27.007608  8584 rpcz_store.cc:276] 0709 13:03:25.727618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.727745 (+   127us) service_pool.cc:224] Handling call
0709 13:03:25.740076 (+ 12331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:26.533892 (+793816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.992263 (+458371us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54002d73d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:26.992525 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1772678016}
I20260709 13:03:27.016570  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:26.984759 13325 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.016865  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:27.026266  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.027357 13325 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.027509  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.028203  9331 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:27.028206  8584 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:27.024847 13510 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.029863 13510 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.029954  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.033432  8584 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:26.979120 13473 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:26.986363 13236 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:27.022454  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:27.038432  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:27.039167  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.040452  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:27.041203  9324 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:26.997677 13156 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:27.001701 13248 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:27.047086  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.013830 13242 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:27.014915 13471 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:27.018420 13488 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:27.035877 13489 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.035727 13476 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.060781  8658 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
W20260709 13:03:27.094298  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44690us
I20260709 13:03:27.128571  8579 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: [8.8-11.10]
I20260709 13:03:27.049443 13404 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:27.135267  8579 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:03:27.018226 13431 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.021139 13495 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.044024 13325 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:27.034971 13510 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:27.113570 13524 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 13:03:27.125854 13527 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:03:27.047801 13283 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:27.126585 13523 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
I20260709 13:03:27.206247  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:27.109354 13534 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
I20260709 13:03:27.133108 13476 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.300769  9324 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:27.274925 13404 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:27.158525  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.275329  8575 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:27.044435 13473 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.134032 13242 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:27.133572 13399 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.169s
I20260709 13:03:27.314045  8579 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:27.302845  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.220654 13325 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:27.323770 13535 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.261s	user 0.000s	sys 0.000s
I20260709 13:03:27.048177 13018 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.325284 13431 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.325732 13504 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.327428 13495 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.359917 13018 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:27.359973 13504 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.362676 13504 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:27.320583 13539 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20260709 13:03:27.328071  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:27.329326 13525 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 13:03:27.329655 13494 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:27.299474 13426 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:27.113044 13248 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.329366  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1089) took 1624 ms. Trace:
I20260709 13:03:27.048805 13474 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.388403  9322 rpcz_store.cc:276] 0709 13:03:25.705041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.705268 (+   227us) service_pool.cc:224] Handling call
0709 13:03:25.705593 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:26.433196 (+727603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.329086 (+895890us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b54003fff54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:27.329294 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1671895424}
I20260709 13:03:27.307085 13283 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:27.306191 13383 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:27.063318 13485 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:27.335827 13341 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.700s	user 0.009s	sys 0.005s
I20260709 13:03:27.338780  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.337286 13473 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.339349 13404 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.334937 13325 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:27.347899  8579 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
W20260709 13:03:27.348152 13282 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.841s	user 0.006s	sys 0.002s
I20260709 13:03:27.350623 13242 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:27.413708 13528 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.359s	user 0.001s	sys 0.000s
I20260709 13:03:27.151209 13489 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.416710 13476 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.417869 13489 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.418856  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2793) took 1062 ms. Trace:
I20260709 13:03:27.419332  8583 rpcz_store.cc:276] 0709 13:03:26.356930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.357081 (+   151us) service_pool.cc:224] Handling call
0709 13:03:26.357465 (+   384us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:26.357821 (+   356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:26.357853 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:26.504281 (+146428us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b2c000014c8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:26.504857 (+   576us) log.cc:844] Serialized 52 byte log entry
0709 13:03:26.713190 (+208333us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:27.023302 (+310112us) log.cc:844] Serialized 14 byte log entry
0709 13:03:27.043621 (+ 20319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.043714 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:27.418530 (+374816us) raft_consensus.cc:1685] finished
0709 13:03:27.418631 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:27.418794 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20374,"prepare.queue_time_us":395089,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":25011,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":31142,"spinlock_wait_cycles":4898048,"thread_start_us":517660,"threads_started":2,"wal-append.queue_time_us":509261}
W20260709 13:03:27.421092  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 548352us
I20260709 13:03:27.421448  9323 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:27.423089 13352 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:27.423502 13352 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:27.423874 13352 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:27.422399  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:27.425693 13485 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:27.427716 13158 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:27.374310 13492 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:27.432365  8582 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [7.7-9.9]
W20260709 13:03:27.379519  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 215841us
I20260709 13:03:27.299459 13510 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:03:27.379619  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:27.379011 13504 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:27.408252 13497 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:27.411223 13533 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 13:03:27.453584  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:27.430935 13248 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:27.402372 13341 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.455508 13497 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:27.439131 13282 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:27.479480  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2645) took 1798 ms (client timeout 2115 ms). Trace:
W20260709 13:03:27.497001  8728 rpcz_store.cc:269] 0709 13:03:25.680872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.681015 (+   143us) service_pool.cc:224] Handling call
0709 13:03:27.479230 (+1798215us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400338e54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:27.479405 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2449484160}
I20260709 13:03:27.499155  8726 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:27.451435 13489 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:03:27.450160 13492 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.457064 13504 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.552968 13547 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.553673 13547 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.556318 13547 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.558130 13156 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:27.484289 13283 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.582173 13352 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:27.588897 13452 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 2.386s	user 0.006s	sys 0.007s
I20260709 13:03:27.586652  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:27.595854  8651 raft_consensus.cc:2368] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14.
I20260709 13:03:27.591864 13478 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:27.599077 13478 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:27.599735 13494 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:27.611244  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 157755us
I20260709 13:03:27.612289  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:27.616580 13548 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.617425 13548 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.621021 13548 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:27.623101  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:27.631790  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:27.634665  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.635600 13474 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:27.636132 13474 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:27.637178 13474 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:27.638850 13522 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.642s	user 0.002s	sys 0.000s
I20260709 13:03:27.640012 13304 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:27.643217  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 775) took 1444 ms. Trace:
I20260709 13:03:27.643656  8650 rpcz_store.cc:276] 0709 13:03:26.198309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.230524 (+ 32215us) service_pool.cc:224] Handling call
0709 13:03:26.248907 (+ 18383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:26.465456 (+216549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.642969 (+1177513us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b5400410854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:27.643150 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":316924288}
I20260709 13:03:27.625207 13247 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:27.653750 13282 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.624652 13494 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.654629  9324 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:27.655579  9326 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:03:27.655684  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:27.659435 13488 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.660180 13492 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.661729  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.605173 13426 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 13:03:27.664875  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.665675  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.672272  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.684796  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:27.664999 13348 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:27.677146 13548 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:27.675407 13341 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:27.714882  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:27.699038 13495 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 13:03:27.693921 13348 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:27.697579 13504 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.739619 13348 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:27.740286 13549 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 13:03:27.749565  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2796) took 1388 ms. Trace:
I20260709 13:03:27.761278  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1092) took 1137 ms. Trace:
I20260709 13:03:27.782759  8575 rpcz_store.cc:276] 0709 13:03:26.361505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.361647 (+   142us) service_pool.cc:224] Handling call
0709 13:03:26.387541 (+ 25894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:27.316513 (+928972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.749238 (+432725us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b540030be54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:27.749486 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2041041664}
W20260709 13:03:27.762964 13531 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.846s	user 0.001s	sys 0.001s
I20260709 13:03:27.768019 13466 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:27.790036  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:27.670630 13400 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 13:03:27.768623 13470 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:27.807850 13470 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.687927 13404 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:03:27.768952 13446 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.769459 13340 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:27.773785  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:27.678992 13488 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.776093  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:27.776582 13325 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.777675  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:27.822114  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:27.777894 13397 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.779863 13348 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:27.834582  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.835343  8579 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:27.778439  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:27.782146  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2391) took 1559 ms. Trace:
I20260709 13:03:27.782851  9329 rpcz_store.cc:276] 0709 13:03:26.623746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.623877 (+   131us) service_pool.cc:224] Handling call
0709 13:03:26.650162 (+ 26285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:26.663793 (+ 13631us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:26.694344 (+ 30551us) log.cc:844] Serialized 14 byte log entry
0709 13:03:26.993874 (+299530us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:26.993946 (+    72us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:26.994018 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.760408 (+766390us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b1800460900. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419
0709 13:03:27.760994 (+   586us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b54002148d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:27.761046 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:27.761201 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91715,"prepare.queue_time_us":32122,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":2866,"raft.queue_time_us":114420,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":9202,"spinlock_wait_cycles":40594048,"thread_start_us":184692,"threads_started":1,"wal-append.queue_time_us":325704}
I20260709 13:03:27.785923 13366 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:27.786310  8651 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:27.786746 12969 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.788879  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2804) took 1243 ms. Trace:
I20260709 13:03:27.733650 13518 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:27.737293 13548 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:27.789662  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:27.793073  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:27.768513 13494 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.806022 13467 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:27.806771 13302 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:27.805377 13466 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.811805  9324 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:27.813385  8652 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:27.860196  8582 raft_consensus.cc:2410] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 13 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 13:03:27.816598  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:27.826731  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.839533  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.842161  8580 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:27.863858  8657 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:27.864933  9331 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:27.865460  8583 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:27.866361  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1124) took 1430 ms. Trace:
I20260709 13:03:27.866796  9327 rpcz_store.cc:276] 0709 13:03:26.435906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.436028 (+   122us) service_pool.cc:224] Handling call
0709 13:03:26.460162 (+ 24134us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:26.466901 (+  6739us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:26.466932 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:26.475143 (+  8211us) log.cc:844] Serialized 26 byte log entry
0709 13:03:27.649341 (+1174198us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b180041b3c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:27.847173 (+197832us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:27.847239 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.847342 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:27.847408 (+    66us) raft_consensus.cc:1685] finished
0709 13:03:27.847428 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:27.866295 (+ 18867us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18174,"raft.queue_time_us":8794,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":22380,"spinlock_wait_cycles":35854720,"thread_start_us":197574,"threads_started":1,"wal-append.queue_time_us":1337270}
I20260709 13:03:27.868996  8579 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:27.841908  8577 rpcz_store.cc:276] 0709 13:03:26.545845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.546018 (+   173us) service_pool.cc:224] Handling call
0709 13:03:27.788672 (+1242654us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54002c9cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:27.788823 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":569718528}
I20260709 13:03:27.842283  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:27.850266  8575 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:03:27.784718 13386 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:27.885114  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2378) took 2679 ms. Trace:
I20260709 13:03:27.838037  8729 rpcz_store.cc:276] 0709 13:03:26.222525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.424056 (+201531us) service_pool.cc:224] Handling call
0709 13:03:26.424434 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:26.424820 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:26.425094 (+   274us) log.cc:844] Serialized 14 byte log entry
0709 13:03:26.879332 (+454238us) log.cc:844] Serialized 14 byte log entry
0709 13:03:27.771507 (+892175us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:27.772023 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 13:03:27.772184 (+   161us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:27.772226 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.772316 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:27.781949 (+  9633us) raft_consensus.cc:1685] finished
0709 13:03:27.781967 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:27.782089 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":903347,"prepare.queue_time_us":85180,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":2670,"raft.queue_time_us":1928803,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":5230,"thread_start_us":414795,"threads_started":2,"wal-append.queue_time_us":709686}
I20260709 13:03:27.814599  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:27.885708  8731 rpcz_store.cc:276] 0709 13:03:25.205790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.227835 (+ 22045us) service_pool.cc:224] Handling call
0709 13:03:26.793613 (+1565778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:26.841006 (+ 47393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.884857 (+1043851us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b5400339350. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:27.885036 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3599977344}
I20260709 13:03:27.890960  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 782) took 1182 ms. Trace:
I20260709 13:03:27.891438  8659 rpcz_store.cc:276] 0709 13:03:26.779742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.786992 (+1007250us) service_pool.cc:224] Handling call
0709 13:03:27.890894 (+103902us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:27.894609  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:27.896658  9326 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 13:03:27.897329  9326 pending_rounds.cc:85] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 8
I20260709 13:03:27.897364  9324 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:27.897810  9326 pending_rounds.cc:107] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:03:27.898216  9326 raft_consensus.cc:2889] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:27.940467  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44912.8us
W20260709 13:03:27.946650  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 74345.9us
I20260709 13:03:27.948995  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:27.845089 13338 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 13:03:27.840989 13446 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.940097 12975 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.018s
I20260709 13:03:27.960522 13488 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:27.963819 13446 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:27.848516 12969 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.965763  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:27.966387  9326 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:27.969584 13548 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.970468 13400 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.970994 13400 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:27.972973 13400 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.899236 13497 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.974355 13325 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:27.898818 13559 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.085s
I20260709 13:03:27.976658  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:27.884057 13518 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:27.982016  8654 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:27.841269 13340 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.984333  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:27.991712  8651 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 7.
I20260709 13:03:27.848129 13366 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:27.985715  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 151941us
I20260709 13:03:27.986958  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:27.941391 13417 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:27.994390  8580 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 13:03:28.002014  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.943511 13347 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.950702 13246 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:27.977758 13531 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:03:27.872639 13467 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:27.952025 13474 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.986122 13446 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.978931 13470 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:27.988365 13539 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 13:03:27.894920 13386 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.001185 13302 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:28.068727  9323 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:28.008204 13488 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:28.085834  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:28.007838 13417 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:28.016824 13331 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.105s
I20260709 13:03:28.005769 13518 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.033996 13341 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:28.007777 13504 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:28.096719  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:28.296917  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 310071us
I20260709 13:03:28.016546 13387 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:28.269960  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2401) took 1664 ms. Trace:
I20260709 13:03:28.055670  8650 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:28.171095  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2782) took 3015 ms. Trace:
I20260709 13:03:28.305011  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:28.271085  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:28.275231 13498 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:28.333118  8654 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:28.333349  8651 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
W20260709 13:03:28.151144 13531 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:03:28.332870  8578 rpcz_store.cc:276] 0709 13:03:25.134737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.300075 (+165338us) service_pool.cc:224] Handling call
0709 13:03:25.300479 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:26.619414 (+1318935us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.149950 (+1530536us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001680d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:28.150153 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3023146368}
I20260709 13:03:28.268740 13466 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.338395  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2399) took 1751 ms. Trace:
I20260709 13:03:28.137830 13569 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:28.107151 13519 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.089s	user 0.004s	sys 0.000s
W20260709 13:03:28.109741 13565 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
I20260709 13:03:28.086068 13485 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:28.304791 13248 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.270908 13366 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.037228 13347 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.305514 13488 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:28.338943  8725 rpcz_store.cc:276] 0709 13:03:26.587224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.587715 (+   491us) service_pool.cc:224] Handling call
0709 13:03:26.620317 (+ 32602us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:26.620692 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:26.620719 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:26.623940 (+  3221us) log.cc:844] Serialized 52 byte log entry
0709 13:03:26.713587 (+ 89647us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:26.713937 (+   350us) log.cc:844] Serialized 14 byte log entry
0709 13:03:26.742689 (+ 28752us) log.cc:844] Serialized 14 byte log entry
0709 13:03:27.759347 (+1016658us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b1800281320. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:03:27.759591 (+   244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.759684 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:27.762361 (+  2677us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:28.278026 (+515665us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b1800281320. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:28.338171 (+ 60145us) raft_consensus.cc:1685] finished
0709 13:03:28.338192 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:28.338330 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31570,"prepare.queue_time_us":1874,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":82,"raft.queue_time_us":1058498,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":27000,"spinlock_wait_cycles":51731840,"thread_start_us":76261,"threads_started":1,"wal-append.queue_time_us":156356}
I20260709 13:03:28.332878  8726 rpcz_store.cc:276] 0709 13:03:26.605830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.605966 (+   136us) service_pool.cc:224] Handling call
0709 13:03:26.606260 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:27.499700 (+893440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.269714 (+770014us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54001e5fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:28.269898 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2052858112}
I20260709 13:03:28.343128  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:28.341375  8655 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:28.085901 13247 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.345872  8580 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:28.269855 12969 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.275746 13325 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.297327 13446 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.275363 13548 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:28.348047  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:28.221624 13467 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.041141 13246 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.349632  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.377556  8731 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 7.
I20260709 13:03:28.353766  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:28.354547  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:28.336094 13387 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:28.355607  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:28.303493 13302 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.363915  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:28.361104 13488 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:28.385757  8651 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:03:28.342276  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:28.340338 13504 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:28.370373  8625 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.809s (SENT)
I20260709 13:03:28.390079  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:28.090552 13474 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.356241 13340 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.370580  8577 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:28.275823 13478 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:28.388902  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:28.387507  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:28.371783 13497 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.361284 13470 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 13:03:28.349030 13569 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.402022  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:28.393486  8654 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260709 13:03:28.378295 13383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.006s	sys 0.029s
I20260709 13:03:28.407940  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:28.382128 13400 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 13:03:28.447499  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:28.392045  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:28.381745 13568 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.358500 13248 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.385092 13368 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:28.395335 13328 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:28.436236 13347 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.437156 13386 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.354560 13417 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.438773 13302 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.439131 13352 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:28.439280 13387 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.439842 13504 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:28.366322 13247 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.441989  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:28.372085 13325 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.514273  8731 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:28.516444 13325 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:28.355101 13485 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:28.519472 13485 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:28.402601 13467 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 13:03:28.520697 13467 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:28.521235 13467 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:28.522535 13467 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.524394 13467 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:28.441527 13491 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:28.525813 13491 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.528971 13491 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.433243 13018 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.555096 13018 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:28.555296  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 43825.6us
I20260709 13:03:28.424551 13516 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:28.557852 13516 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:28.558142  8623 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:28.558307 13516 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:28.383122 13548 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:28.561919  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 84704.3us
I20260709 13:03:28.451325  8657 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:03:28.454787  8651 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:28.456627  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:28.587613  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 908) took 1611 ms. Trace:
I20260709 13:03:28.588022  8584 rpcz_store.cc:276] 0709 13:03:26.976147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.046805 (+ 70658us) service_pool.cc:224] Handling call
0709 13:03:28.587425 (+1540620us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b540017a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:28.587558 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320347008}
I20260709 13:03:28.449373  8697 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:28.463855  8652 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:28.465267  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:28.589162  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 139949us
W20260709 13:03:28.589771  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:28.473878 13572 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:03:28.590843  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.591130  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:28.469573  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:28.474879  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:28.481957  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2797) took 2118 ms. Trace:
I20260709 13:03:28.592306  8655 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
W20260709 13:03:28.593571  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:28.467288  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:28.487083 13328 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:28.464524  8658 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:03:28.495930  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:28.462399 12969 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:28.464884 13348 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.619s	user 0.009s	sys 0.001s
I20260709 13:03:28.455168 13331 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:28.463723 13338 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:28.502415  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 115983us
I20260709 13:03:28.553515 13559 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.463600 13341 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:28.463503 13445 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:28.462175 13518 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:28.480283 13390 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:28.454489 13446 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.454236 13246 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:28.462891 13569 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:28.505043  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 70976.8us
I20260709 13:03:28.492327 13368 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:28.433460 13478 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:28.462741 13570 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:28.535276  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:28.670833  8659 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:28.455430 13582 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:03:28.539196 13247 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:26.028537  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:28.487335 13568 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.534355 13387 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:28.462570 13366 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:03:28.564173 13548 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:28.446017 13383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.906s	user 0.006s	sys 0.029s
W20260709 13:03:28.565596  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 173588us
I20260709 13:03:28.569873  8657 pending_rounds.cc:85] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 6
I20260709 13:03:28.586726  8577 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:28.469869  8579 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:03:28.683151  8657 pending_rounds.cc:107] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:28.501061 13428 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 13:03:28.640452  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.587486  8656 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:28.597918  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:28.566028 13018 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.661774  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:28.686930  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 89156.8us
I20260709 13:03:28.662576  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.585870 13491 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:28.656772  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:28.667037  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 917) took 1005 ms. Trace:
I20260709 13:03:28.638387 13280 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.692198  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:28.596432 13485 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.595435 13471 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:28.673914 13569 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:28.672046  8724 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:28.652163 13328 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:28.677435  8726 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:28.653340 13390 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:28.704476  8725 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:28.680954  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.683106  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.683651  8657 raft_consensus.cc:2889] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:28.592070  8581 rpcz_store.cc:276] 0709 13:03:26.363047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.363182 (+   135us) service_pool.cc:224] Handling call
0709 13:03:26.363533 (+   351us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:26.363950 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:26.363983 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:26.384985 (+ 21002us) log.cc:844] Serialized 52 byte log entry
0709 13:03:26.578373 (+193388us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:26.613172 (+ 34799us) log.cc:844] Serialized 14 byte log entry
0709 13:03:26.613417 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.613490 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:27.383343 (+769853us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b18003e1dc0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b29522 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:28.106400 (+723057us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:28.481679 (+375279us) raft_consensus.cc:1685] finished
0709 13:03:28.481716 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:28.481891 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1582,"prepare.queue_time_us":32183,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":3351,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":6362,"spinlock_wait_cycles":42208512,"thread_start_us":192987,"threads_started":1,"wal-append.queue_time_us":537322}
I20260709 13:03:28.685422  9296 leader_election.cc:400] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14.
I20260709 13:03:28.690446  8582 rpcz_store.cc:276] 0709 13:03:27.661292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.661453 (+   161us) service_pool.cc:224] Handling call
0709 13:03:28.666768 (+1005315us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54003a4b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:28.666960 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":424894208}
I20260709 13:03:28.649329 13340 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:28.706732  9296 leader_election.cc:403] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:03:28.706974 13340 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:28.655953 13559 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.657325 13445 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.693825 13548 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:03:28.722592 13429 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.699s	user 0.012s	sys 0.051s
I20260709 13:03:28.655167 13338 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.699532  8728 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:28.700634  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.657910 13341 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.707976 13474 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.708312  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:28.758184  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 49953.9us
I20260709 13:03:28.707616 13340 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:28.677194 13246 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 13:03:28.755223  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.749725  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.750438  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:28.697000  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 201169us
I20260709 13:03:28.755596  8731 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:28.772248  9331 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:28.773156  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.695458  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:28.761866 13584 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:03:28.762116 13341 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.762594  8729 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:28.764393 13466 raft_consensus.cc:1064] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
W20260709 13:03:28.754518 13429 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.747s	user 0.012s	sys 0.051s
I20260709 13:03:28.697184 13488 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.744544 13400 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:28.758921  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.801054  8728 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:28.700026 13485 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.772328  8730 raft_consensus.cc:2410] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 13 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
W20260709 13:03:28.773252  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 102878us
I20260709 13:03:28.743896 13582 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14.
I20260709 13:03:28.803375  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.746500 13331 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 13:03:28.699976 13280 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.702502 13471 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.777807 13338 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:28.791929  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:28.813627  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 107574us
I20260709 13:03:28.794133  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.814016  9328 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:28.819553  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:28.799379  8727 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:28.802094  8653 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 13:03:28.773371 13548 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:28.811400  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:28.806391 13400 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.821375  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:28.821955  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:28.822474 13280 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:28.822921 13400 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:28.823503 13400 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:28.823624 13595 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 13:03:28.824975  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:28.825716  9323 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:28.828393  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:28.829232  9323 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:03:28.812903 13246 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 13:03:28.839027 13445 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.841099 13471 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:28.842710 13125 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.842965 13488 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:28.806859 13473 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.844656 13473 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.845052 13466 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.846890 13473 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:28.748797 13368 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:28.803215 13518 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 13:03:28.851341  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:28.813513 13592 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 13:03:28.874985  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:28.955960  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 318715us
I20260709 13:03:28.862264  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:28.780870 13390 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:28.814581 13504 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.222s
W20260709 13:03:29.124631  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:29.038693  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:29.108147  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.138713  8732 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:03:29.020740  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63245.2us
I20260709 13:03:29.066699  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.016075  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:29.142439  8724 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
W20260709 13:03:29.077997  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:29.306478  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:29.142796  8656 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:29.143782  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:29.122143  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.123871  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:29.308439  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 164783us
I20260709 13:03:29.016831  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:29.269174 13585 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.234879  8659 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:29.085350 13569 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.086539 13491 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.242584 13341 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:29.129663  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:29.243090  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:29.243304 13390 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:28.858289 13586 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:29.243829  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.037216 13445 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.207s
I20260709 13:03:29.035378 13156 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:03:29.246065  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65023.4us
W20260709 13:03:29.246603  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67575.5us
I20260709 13:03:29.270213 13568 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:29.259123 13429 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:29.266592  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:28.972983  9328 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:29.309363  8651 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:29.067178  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.310173  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:29.392823  8730 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:29.314321  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.314548  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:29.317914  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:29.340631  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.373507  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.342182  8575 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:29.316061  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:29.282889  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.343731  8577 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:30.344396  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2217) took 1062 ms. Trace:
I20260709 13:03:30.344743  8577 rpcz_store.cc:276] 0709 13:03:29.282345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.282545 (+   200us) service_pool.cc:224] Handling call
0709 13:03:30.344296 (+1061751us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:29.281087  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:30.427670  8579 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:29.291049  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:29.531584 13612 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:30.141691  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.143534  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.432689  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2816) took 1143 ms. Trace:
I20260709 13:03:30.172240  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.433982  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2822) took 1140 ms. Trace:
I20260709 13:03:30.434376  8581 rpcz_store.cc:276] 0709 13:03:29.293444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.171522 (+878078us) service_pool.cc:224] Handling call
0709 13:03:30.433922 (+262400us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.436007  8581 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:30.436715  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2819) took 1144 ms. Trace:
I20260709 13:03:30.177224  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:30.189288  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.372s	user 0.032s	sys 0.037s
I20260709 13:03:30.438063  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2821) took 1145 ms. Trace:
W20260709 13:03:30.438269  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.621s	user 0.032s	sys 0.039s
I20260709 13:03:30.433050  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2827) took 1066 ms. Trace:
W20260709 13:03:30.438769  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 1.62157e+06us
I20260709 13:03:30.439108  8576 rpcz_store.cc:276] 0709 13:03:29.367406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.142967 (+775561us) service_pool.cc:224] Handling call
0709 13:03:30.432965 (+289998us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.438484  8582 rpcz_store.cc:276] 0709 13:03:29.293043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.339628 (+ 46585us) service_pool.cc:224] Handling call
0709 13:03:30.437966 (+1098338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.442931  8582 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:30.245880  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.925s	user 0.017s	sys 0.037s
I20260709 13:03:30.229286  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.443706  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2826) took 1078 ms. Trace:
I20260709 13:03:30.347122  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.444267  8578 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:30.354514  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2212) took 1994 ms. Trace:
W20260709 13:03:30.364185  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.444717  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2823) took 1151 ms. Trace:
I20260709 13:03:30.445514  8577 rpcz_store.cc:276] 0709 13:03:29.294007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.346677 (+1052670us) service_pool.cc:224] Handling call
0709 13:03:30.444619 (+ 97942us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.335383  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:30.447882  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 798) took 1591 ms (client timeout 1893 ms). Trace:
I20260709 13:03:30.444955  8580 rpcz_store.cc:276] 0709 13:03:28.360653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.382608 (+ 21955us) service_pool.cc:224] Handling call
0709 13:03:28.404680 (+ 22072us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:28.421815 (+ 17135us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:28.421876 (+    61us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:28.433167 (+ 11291us) log.cc:844] Serialized 78 byte log entry
0709 13:03:28.598397 (+165230us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:28.654057 (+ 55660us) log.cc:844] Serialized 14 byte log entry
0709 13:03:28.659873 (+  5816us) log.cc:844] Serialized 14 byte log entry
0709 13:03:28.797487 (+137614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.797571 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:30.055552 (+1257981us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b1800425620. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:30.082930 (+ 27378us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b1800425620. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:30.353958 (+271028us) raft_consensus.cc:1685] finished
0709 13:03:30.353984 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:30.354303 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5280,"prepare.queue_time_us":80199,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":74,"raft.queue_time_us":92991,"raft.run_cpu_time_us":723,"raft.run_wall_time_us":14674,"spinlock_wait_cycles":23512192,"thread_start_us":272583,"threads_started":2,"wal-append.queue_time_us":216967}
W20260709 13:03:30.448259  8658 rpcz_store.cc:269] 0709 13:03:28.856305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.334810 (+1478505us) service_pool.cc:224] Handling call
0709 13:03:30.447794 (+112984us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.449363  8579 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:03:30.449966  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2216) took 1169 ms. Trace:
I20260709 13:03:30.449898  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:29.401646  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 135120us
I20260709 13:03:30.507694 13620 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.231401 13620 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.234280 13620 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:30.511605 13622 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:30.558398 13627 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:31.235311 13622 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.235448 13627 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:30.569715 13624 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:30.617990 13629 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:31.237951 13622 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:31.237794 13629 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.239548 13627 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.240134 13629 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:30.484122  8479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 204) took 31145 ms (client timeout 15000 ms). Trace:
W20260709 13:03:30.443872  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.123s	user 0.018s	sys 0.038s
I20260709 13:03:30.629673  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:31.259709  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.573036  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2825) took 1188 ms. Trace:
I20260709 13:03:31.263417  8576 rpcz_store.cc:276] 0709 13:03:29.364814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.440071 (+1075257us) service_pool.cc:224] Handling call
0709 13:03:30.440692 (+   621us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 13:03:30.452076 (+ 11384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:30.452123 (+    47us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 13:03:30.453528 (+  1405us) log.cc:844] Serialized 104 byte log entry
0709 13:03:30.485275 (+ 31747us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:30.513123 (+ 27848us) log.cc:844] Serialized 14 byte log entry
0709 13:03:30.514690 (+  1567us) log.cc:844] Serialized 14 byte log entry
0709 13:03:30.551379 (+ 36689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.552149 (+   770us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:30.552296 (+   147us) raft_consensus.cc:1685] finished
0709 13:03:30.552312 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:30.552502 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":42580,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":14074,"raft.queue_time_us":176998,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":345,"thread_start_us":77748,"threads_started":3,"wal-append.queue_time_us":40432}
I20260709 13:03:30.444792  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2818) took 1152 ms. Trace:
I20260709 13:03:31.266697  8578 rpcz_store.cc:276] 0709 13:03:29.291931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.228522 (+936591us) service_pool.cc:224] Handling call
0709 13:03:30.444685 (+216163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.268002  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.450426  8579 rpcz_store.cc:276] 0709 13:03:29.280495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.280681 (+   186us) service_pool.cc:224] Handling call
0709 13:03:30.449892 (+1169211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.521361  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.437077  8581 rpcz_store.cc:276] 0709 13:03:29.292309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.435193 (+1142884us) service_pool.cc:224] Handling call
0709 13:03:30.435728 (+   535us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.436442 (+   714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.436631 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.454547  8582 rpcz_store.cc:276] 0709 13:03:29.366038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.442144 (+1076106us) service_pool.cc:224] Handling call
0709 13:03:30.442616 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.443404 (+   788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.443642 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.452231  8583 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:03:31.275686  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2817) took 1984 ms. Trace:
I20260709 13:03:31.276077  8583 rpcz_store.cc:276] 0709 13:03:29.291463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.291652 (+   189us) service_pool.cc:224] Handling call
0709 13:03:29.292072 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:30.451513 (+1159441us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b54000dcb20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:03:31.275173 (+823660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.275598 (+   425us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51544960}
I20260709 13:03:30.433123  8584 rpcz_store.cc:276] 0709 13:03:29.290077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.290694 (+   617us) service_pool.cc:224] Handling call
0709 13:03:30.432494 (+1141800us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.330952  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:29.472703  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.287104  8651 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
W20260709 13:03:31.287807  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 790) took 2432 ms (client timeout 1776 ms). Trace:
W20260709 13:03:31.288159  8651 rpcz_store.cc:269] 0709 13:03:28.854980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.432658 (+577678us) service_pool.cc:224] Handling call
0709 13:03:31.287706 (+1855048us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:30.250083  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 778) took 3544 ms. Trace:
I20260709 13:03:31.290030  8653 rpcz_store.cc:276] 0709 13:03:26.705824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.746306 (+ 40482us) service_pool.cc:224] Handling call
0709 13:03:26.746709 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:28.820617 (+2073908us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.249256 (+1428639us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b5400420c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:30.249817 (+   561us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4726092544}
W20260709 13:03:31.291443  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2604) took 2430 ms (client timeout 2071 ms). Trace:
W20260709 13:03:31.291785  8653 rpcz_store.cc:269] 0709 13:03:28.861371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.291240 (+2429869us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.291365 (+   125us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.292550  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 800) took 1876 ms (client timeout 1517 ms). Trace:
W20260709 13:03:31.292840  8653 rpcz_store.cc:269] 0709 13:03:29.415984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.292385 (+1876401us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.292504 (+   119us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:30.351758  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:30.311120  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:31.296437  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 792) took 2441 ms (client timeout 1909 ms). Trace:
I20260709 13:03:29.382174  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:31.298170  8656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2600) took 2441 ms (client timeout 1599 ms). Trace:
I20260709 13:03:30.184890  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2844) took 1752 ms. Trace:
I20260709 13:03:31.299378  8657 rpcz_store.cc:276] 0709 13:03:28.432468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.432652 (+   184us) service_pool.cc:224] Handling call
0709 13:03:28.450897 (+ 18245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:28.706145 (+255248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.184062 (+1477917us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b540005a0d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:30.184523 (+   461us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27289344}
I20260709 13:03:30.466681  8658 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
W20260709 13:03:31.301721  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 795) took 2445 ms (client timeout 2143 ms). Trace:
I20260709 13:03:29.408656  8659 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:31.303985  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:31.304731  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2602) took 2445 ms (client timeout 1687 ms). Trace:
W20260709 13:03:31.305120  8659 rpcz_store.cc:269] 0709 13:03:28.858813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.317650 (+458837us) service_pool.cc:224] Handling call
0709 13:03:31.304538 (+1986888us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.306103  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2853) took 2061 ms (client timeout 1897 ms). Trace:
W20260709 13:03:31.306434  8659 rpcz_store.cc:269] 0709 13:03:29.244834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.305872 (+2061038us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.306034 (+   162us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.306952  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:31.307193  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2613) took 1939 ms (client timeout 1934 ms). Trace:
W20260709 13:03:31.307541  8659 rpcz_store.cc:269] 0709 13:03:29.369432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.307028 (+1937596us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.307153 (+   125us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.308413  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:30.305065  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:31.311043  8652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2601) took 2454 ms (client timeout 1813 ms). Trace:
W20260709 13:03:31.311401  8652 rpcz_store.cc:269] 0709 13:03:28.856893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.304394 (+1447501us) service_pool.cc:224] Handling call
0709 13:03:31.310967 (+1006573us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.312404  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:29.952315  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.003s	sys 0.011s
W20260709 13:03:31.318266  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.943s	user 0.006s	sys 0.011s
W20260709 13:03:31.318746  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 2.1889e+06us
W20260709 13:03:29.397672  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 83188.8us
I20260709 13:03:31.322734  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:30.117141  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:29.424196  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:29.394658  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:29.417755  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:31.332454  8726 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 13
W20260709 13:03:31.333437  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2416) took 1946 ms (client timeout 1612 ms). Trace:
I20260709 13:03:29.399019  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:31.343037  8732 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
W20260709 13:03:31.343797  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2413) took 2017 ms (client timeout 1764 ms). Trace:
W20260709 13:03:31.344297  8732 rpcz_store.cc:269] 0709 13:03:29.326215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.371577 (+ 45362us) service_pool.cc:224] Handling call
0709 13:03:31.343678 (+1972101us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:30.302394  8729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 897) took 1662 ms (client timeout 1748 ms). Trace:
I20260709 13:03:29.423056  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:30.270319  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:29.397506  9297 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.570s (SENT)
I20260709 13:03:31.351612  9297 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:30.327127  9322 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:31.353875  9322 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:31.355289  9322 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.357154  9322 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:30.621380  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1117) took 1853 ms. Trace:
I20260709 13:03:30.283856  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1138) took 1466 ms. Trace:
I20260709 13:03:31.368049  9326 rpcz_store.cc:276] 0709 13:03:28.768288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.768448 (+   160us) service_pool.cc:224] Handling call
0709 13:03:28.771332 (+  2884us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:28.771673 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:28.791002 (+ 19329us) log.cc:844] Serialized 14 byte log entry
0709 13:03:30.372914 (+1581912us) spinlock_profiling.cc:243] Waited 7.61 ms on lock 0x7b18004032a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:03:30.617245 (+244331us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:30.618473 (+  1228us) log.cc:844] Serialized 26 byte log entry
0709 13:03:30.618640 (+   167us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:30.618662 (+    22us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.618749 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:30.620905 (+  2156us) raft_consensus.cc:1685] finished
0709 13:03:30.621000 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:30.621195 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":687,"prepare.queue_time_us":13116,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":20352,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":3219,"spinlock_wait_cycles":17508608,"thread_start_us":243711,"threads_started":2,"wal-append.queue_time_us":1605795}
I20260709 13:03:31.374686  9331 rpcz_store.cc:276] 0709 13:03:28.817510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.817903 (+   393us) service_pool.cc:224] Handling call
0709 13:03:28.818549 (+   646us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:28.818904 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:28.866374 (+ 47470us) log.cc:844] Serialized 14 byte log entry
0709 13:03:30.282555 (+1416181us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b1800444880. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a
0709 13:03:30.283004 (+   449us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:30.283099 (+    95us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:30.283244 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.283380 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:30.283605 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":245879,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":25404,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":350,"spinlock_wait_cycles":50812160,"thread_start_us":218068,"threads_started":1,"wal-append.queue_time_us":453395,"wal-append.run_cpu_time_us":2550,"wal-append.run_wall_time_us":1283088}
I20260709 13:03:29.402216 12171 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:31.380640 12171 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.381088 12171 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:29.979840 13125 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.388345 13125 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:29.401492 13156 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:31.413087 13156 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:29.452299 13158 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:31.418292 13158 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.420632 13158 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:30.369028 13183 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.422087 13183 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:29.981658 13236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.002s	sys 0.002s
W20260709 13:03:31.427150 13236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.052s	user 0.005s	sys 0.002s
I20260709 13:03:30.301515 13247 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.085217 13302 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.187s
I20260709 13:03:31.431073 13183 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:30.063036 13328 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.317s	user 0.009s	sys 0.009s
I20260709 13:03:31.446496 13328 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:30.131711 13340 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.350s	user 0.002s	sys 0.024s
I20260709 13:03:31.451084 13340 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:29.390826 13341 consensus_queue.cc:1243] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:03:29.501706 13368 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.005s	sys 0.044s
W20260709 13:03:29.496892 13387 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.813s	user 0.003s	sys 0.007s
I20260709 13:03:30.363903 13397 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:31.470651  8731 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:31.471048 13397 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:30.326771 13400 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:31.474356 13644 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.474506 13400 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.475049 13644 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.478592  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:30.326512 13405 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:31.482483 13405 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:30.022691 13445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.005s	sys 0.004s
W20260709 13:03:31.487749 13445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.098s	user 0.008s	sys 0.004s
W20260709 13:03:30.016369 13466 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.001s	sys 0.001s
W20260709 13:03:31.490352 13466 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.982s	user 0.003s	sys 0.001s
I20260709 13:03:31.495214 13611 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:31.496012 13611 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:29.502496 13467 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.978s	user 0.010s	sys 0.004s
W20260709 13:03:29.499003 13478 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.826s	user 0.010s	sys 0.001s
I20260709 13:03:31.023844 13495 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:31.506232 13495 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.509663 13473 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.510555 13473 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:29.306741 13248 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:30.060251 13390 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.694s	user 0.005s	sys 0.021s
I20260709 13:03:30.037256 13429 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.522563 13429 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:30.536847 13498 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:03:30.069329 13516 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.510s	user 0.012s	sys 0.008s
I20260709 13:03:31.534413 13516 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:30.075249 13417 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:30.050067 13428 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:31.539811 13428 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:31.541004 13428 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.541865  8733 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:31.542115 13428 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:31.543390  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 902) took 2864 ms (client timeout 2448 ms). Trace:
W20260709 13:03:31.543848  8733 rpcz_store.cc:269] 0709 13:03:28.678559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.678868 (+   309us) service_pool.cc:224] Handling call
0709 13:03:31.543097 (+2864229us) spinlock_profiling.cc:243] Waited 2.77 s on lock 0x7b5400416254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:31.543317 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6363079424}
I20260709 13:03:30.076128 13446 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:31.546468 13368 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.696s	user 0.009s	sys 0.045s
I20260709 13:03:31.546720 13446 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.548103 13397 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:31.548988 13446 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:30.511816 13468 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:31.550249 13468 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:31.552300  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.552484 13468 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:30.301566 13471 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.553824 13471 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.556130 13471 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:30.064987 13485 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:29.363521 13491 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.560803 13491 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:29.319927 13585 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.562295 13400 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.563136 13429 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:31.563773 13585 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:29.337999 13497 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:30.029793 13599 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:31.571130 13599 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:30.248194 13559 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:29.343295 13569 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.578866 13569 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:30.079181 13580 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.634s	user 0.003s	sys 0.013s
W20260709 13:03:30.249670 13598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.003s	sys 0.000s
W20260709 13:03:29.521999 13603 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 13:03:31.584467  8727 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:30.632962 13612 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:31.588263 13612 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:30.326467 13618 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:31.590762 13618 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.034487 13597 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:29.364292 13586 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.596081 13586 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:31.597846 13612 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:31.599094 13618 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:29.467387 13593 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:31.601317 13593 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.601800 13593 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:29.315018 13595 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:31.173781 13631 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:31.176514  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.607543 13631 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.610357 13631 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:30.475630  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:31.616398  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 1.14101e+06us
I20260709 13:03:31.246254 13624 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.619840 13624 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:31.243024  8479 rpcz_store.cc:269] 0709 13:02:59.339017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:59.341605 (+  2588us) service_pool.cc:224] Handling call
0709 13:03:30.483787 (+31142182us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:20.840143 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:20.841001 (+   858us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:20.841067 (+    66us) write_op.cc:435] Acquired schema lock
0709 13:03:20.841096 (+    29us) tablet.cc:662] Decoding operations
0709 13:03:20.843123 (+  2027us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:20.843268 (+   145us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:20.843309 (+    41us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:03:21.052906 (+209597us) tablet.cc:701] Row locks acquired
0709 13:03:21.066363 (+ 13457us) write_op.cc:260] PREPARE: finished
0709 13:03:21.106477 (+ 40114us) write_op.cc:270] Start()
0709 13:03:21.133182 (+ 26705us) write_op.cc:276] Timestamp: P: 1783602201096412 usec, L: 0
0709 13:03:21.139312 (+  6130us) op_driver.cc:348] REPLICATION: starting
0709 13:03:21.166392 (+ 27080us) log.cc:844] Serialized 6178 byte log entry
0709 13:03:21.397047 (+230655us) op_driver.cc:464] REPLICATION: finished
0709 13:03:21.398731 (+  1684us) write_op.cc:301] APPLY: starting
0709 13:03:21.398848 (+   117us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:21.904151 (+505303us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:21.904292 (+   141us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:22.694412 (+790120us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b180038a940. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14099550ec 00007f14099a1c0a 00007f14099a14f7 00007f14099a1490 00007f1409958f48 00007f140d66d1bb 00007f140d6b45d4 00007f140d5608da 00007f140d5633cb
0709 13:03:22.988079 (+293667us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:23.130258 (+142179us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 13:03:23.133387 (+  3129us) write_op.cc:312] APPLY: finished
0709 13:03:23.151010 (+ 17623us) log.cc:844] Serialized 148 byte log entry
0709 13:03:23.234136 (+ 83126us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:23.241765 (+  7629us) write_op.cc:454] Released schema lock
0709 13:03:23.254694 (+ 12929us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:23.287648 (+ 32954us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:23.287688 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":425595,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":1388,"apply.run_cpu_time_us":41952,"apply.run_wall_time_us":1882731,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70975,"num_ops":16,"prepare.queue_time_us":202844,"prepare.run_cpu_time_us":62137,"prepare.run_wall_time_us":451326,"raft.queue_time_us":50031,"raft.run_cpu_time_us":1291,"raft.run_wall_time_us":2380,"replication_time_us":257560,"spinlock_wait_cycles":6062336,"thread_start_us":383559,"threads_started":5,"wal-append.queue_time_us":209066,"wal-append.run_cpu_time_us":13025,"wal-append.run_wall_time_us":2158668}]]}
W20260709 13:03:31.296836  8654 rpcz_store.cc:269] 0709 13:03:28.855331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.351182 (+1495851us) service_pool.cc:224] Handling call
0709 13:03:31.296350 (+945168us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.622597  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2609) took 2301 ms (client timeout 2216 ms). Trace:
W20260709 13:03:31.623070  8654 rpcz_store.cc:269] 0709 13:03:29.321488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.622335 (+2300847us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.622521 (+   186us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.624313  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.625447 13593 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:31.296859  8655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2605) took 2422 ms (client timeout 1827 ms). Trace:
W20260709 13:03:31.300602  8550 connection.cc:256] RPC call timeout handler was delayed by -1.9826411519999283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
W20260709 13:03:31.626708  8550 connection.cc:256] RPC call timeout handler was delayed by -1.9462036250000665s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
I20260709 13:03:31.626973 13559 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:31.627573 13559 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.280151  8477 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:31.628250 13559 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:31.618440 13586 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:31.298529  8656 rpcz_store.cc:269] 0709 13:03:28.856744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.317961 (+461217us) service_pool.cc:224] Handling call
0709 13:03:31.297976 (+1980015us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.302141  8653 rpcz_store.cc:269] 0709 13:03:28.855836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.301365 (+2445529us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.301546 (+   181us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.626499  8655 rpcz_store.cc:269] 0709 13:03:28.874289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.310564 (+1436275us) service_pool.cc:224] Handling call
0709 13:03:31.296799 (+986235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.631836  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.632194  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.632750  8656 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:31.633046  8655 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:31.633240  8656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2599) took 2776 ms (client timeout 3203 ms). Trace:
W20260709 13:03:31.633674  8656 rpcz_store.cc:269] 0709 13:03:28.856599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.631456 (+2774857us) service_pool.cc:224] Handling call
0709 13:03:31.633182 (+  1726us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9984}
I20260709 13:03:31.633982  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.634701  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.635269  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.635886  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:31.636065  8655 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:31.636497  8656 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:31.315050  8579 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 13:03:31.316063  8657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2606) took 2355 ms (client timeout 2162 ms). Trace:
I20260709 13:03:31.313339  8652 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260709 13:03:31.309170  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2607) took 2223 ms (client timeout 2238 ms). Trace:
I20260709 13:03:31.260993  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:31.332130  8725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2674) took 2221 ms (client timeout 2129 ms). Trace:
W20260709 13:03:31.333874  8730 rpcz_store.cc:269] 0709 13:03:29.387225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.393648 (+  6423us) service_pool.cc:224] Handling call
0709 13:03:31.333217 (+1939569us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.341362  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:31.346125  8729 rpcz_store.cc:269] 0709 13:03:28.640038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.640209 (+   171us) service_pool.cc:224] Handling call
0709 13:03:30.301923 (+1661714us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b5400204ed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:30.302181 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149472128}
W20260709 13:03:31.349321  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.926e+06us
I20260709 13:03:31.350337 13629 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:30.203074  9325 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:31.357797 13622 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:31.359035  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 1.96132e+06us
W20260709 13:03:31.360503  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2682) took 1873 ms (client timeout 1687 ms). Trace:
I20260709 13:03:31.369599  8726 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
W20260709 13:03:31.377491  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67861.6us
I20260709 13:03:31.379215 13640 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:31.397882 12171 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.396937 13638 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:31.390323 13630 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:31.402360  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08002d4160 after lost signal to thread 7484
W20260709 13:03:31.409153  8699 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:40420 dropped due to backpressure: service queue is full with 50 items
I20260709 13:03:31.409621 13633 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:31.411219 13125 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.417184  9322 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 13:03:31.429503 13247 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:31.416559  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54177.4us
W20260709 13:03:31.442546  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 120110us
I20260709 13:03:31.457881 13643 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:31.458127  8728 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:31.460680 13340 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.466739 13328 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.467638 13568 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:31.473693 13639 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:31.473078  8731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2647) took 2706 ms (client timeout 1724 ms). Trace:
I20260709 13:03:31.483443 13644 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:31.517764  8724 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:03:31.560863 13473 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.561518 13405 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:31.563916 13611 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.564960 13516 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:30.087751 13576 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.719s	user 0.018s	sys 0.000s
I20260709 13:03:31.566030 13497 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.567579 13495 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:31.570362  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2424) took 2134 ms (client timeout 1757 ms). Trace:
W20260709 13:03:31.574076  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.579178 13242 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.777052 13617 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.777254 13242 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:31.583607 13598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.183s	user 0.004s	sys 0.000s
W20260709 13:03:31.586166  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2673) took 2487 ms (client timeout 2115 ms). Trace:
I20260709 13:03:31.589761 13599 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:31.783299  8727 rpcz_store.cc:269] 0709 13:03:29.098311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.373218 (+274907us) service_pool.cc:224] Handling call
0709 13:03:31.585855 (+2212637us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b5400338e54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:31.586075 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5035276032}
I20260709 13:03:31.594538 13597 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.786410 13597 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:31.627650  8550 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.827s (SENT) [suppressed 40 similar messages]
W20260709 13:03:31.289299  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 799) took 2432 ms (client timeout 2057 ms). Trace:
W20260709 13:03:31.787626  8651 rpcz_store.cc:269] 0709 13:03:28.856450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.289072 (+2432622us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.289233 (+   161us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.788832  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.789785  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2608) took 2663 ms. Trace:
I20260709 13:03:31.790139  8651 rpcz_store.cc:276] 0709 13:03:29.126361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.788430 (+2662069us) service_pool.cc:224] Handling call
0709 13:03:31.789718 (+  1288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.791121  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:31.791796  8651 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:31.629721  8477 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:31.794502 13640 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.795178  8477 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 7, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 7 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:03:31.302141  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2603) took 2441 ms (client timeout 1928 ms). Trace:
W20260709 13:03:31.797669  8652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2614) took 2423 ms (client timeout 2026 ms). Trace:
W20260709 13:03:31.797583  8658 rpcz_store.cc:269] 0709 13:03:28.861098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.449449 (+1588351us) service_pool.cc:224] Handling call
0709 13:03:31.302067 (+852618us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.798154  8652 rpcz_store.cc:269] 0709 13:03:29.375022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.312084 (+1937062us) service_pool.cc:224] Handling call
0709 13:03:31.797568 (+485484us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.799024  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.631860  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.801898  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2854) took 2556 ms. Trace:
I20260709 13:03:31.802278  8653 rpcz_store.cc:276] 0709 13:03:29.245152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.800778 (+2555626us) service_pool.cc:224] Handling call
0709 13:03:31.801306 (+   528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.801573 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.801598 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.801641 (+    43us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:31.801673 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.801726 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.801819 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.797102  8477 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 8. New cstate: current_term: 8 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:31.765816  9322 pending_rounds.cc:85] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 9
W20260709 13:03:31.713102  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:31.767040  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08002f5d00 after lost signal to thread 8516
W20260709 13:03:31.766006  8657 rpcz_store.cc:269] 0709 13:03:28.960753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.315829 (+2355076us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.315978 (+   149us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.637320  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:31.768049  9325 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:31.766376  8731 rpcz_store.cc:269] 0709 13:03:28.766221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.772931 (+  6710us) service_pool.cc:224] Handling call
0709 13:03:31.472816 (+2699885us) spinlock_profiling.cc:243] Waited 2.7 s on lock 0x7b54000893d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:31.472983 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6200866688}
I20260709 13:03:31.807631  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2225) took 1243 ms. Trace:
W20260709 13:03:31.767283  8659 rpcz_store.cc:269] 0709 13:03:29.086068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.308135 (+2222067us) service_pool.cc:224] Handling call
0709 13:03:31.309106 (+   971us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.775211  8733 rpcz_store.cc:269] 0709 13:03:29.435917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.570157 (+2134240us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.570279 (+   122us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.773332  8726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2679) took 2382 ms (client timeout 1781 ms). Trace:
W20260709 13:03:31.776160  8732 rpcz_store.cc:269] 0709 13:03:29.487515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.360246 (+1872731us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.360410 (+   164us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.773761  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2226) took 1203 ms. Trace:
W20260709 13:03:31.637442  8654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 796) took 2769 ms (client timeout 3001 ms). Trace:
W20260709 13:03:31.809612  8654 rpcz_store.cc:269] 0709 13:03:28.856018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.623900 (+2767882us) service_pool.cc:224] Handling call
0709 13:03:31.625275 (+  1375us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.777736 13617 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.779316  8584 raft_consensus.cc:2410] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
W20260709 13:03:31.779537  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2654) took 3008 ms (client timeout 2334 ms). Trace:
W20260709 13:03:31.810910  8724 rpcz_store.cc:269] 0709 13:03:28.770760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.309877 (+539117us) service_pool.cc:224] Handling call
0709 13:03:31.779283 (+2469406us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b5400203ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:31.779473 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4885667072}
I20260709 13:03:31.779793  8575 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:31.811785  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2610) took 2479 ms. Trace:
I20260709 13:03:31.780582 13497 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:31.781369 13247 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:31.781967 13242 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.812218  8657 rpcz_store.cc:276] 0709 13:03:29.332787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.810877 (+2478090us) service_pool.cc:224] Handling call
0709 13:03:31.811239 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.811446 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.811472 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.811515 (+    43us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:31.811556 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.811612 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.811714 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.813294  8605 debug-util.cc:398] Leaking SignalData structure 0x7b080002fdc0 after lost signal to thread 8608
W20260709 13:03:31.813726  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08002e3fa0 after lost signal to thread 8609
I20260709 13:03:31.814951  8655 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 13:03:31.814988  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcee0 after lost signal to thread 8683
I20260709 13:03:31.815227  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2612) took 2450 ms. Trace:
I20260709 13:03:31.815863  8653 rpcz_store.cc:276] 0709 13:03:29.365686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.814177 (+2448491us) service_pool.cc:224] Handling call
0709 13:03:31.814586 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.814855 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.814881 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.814937 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:31.814992 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.815049 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.815150 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48128}
I20260709 13:03:31.816464  8655 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 13:03:31.816639  8605 debug-util.cc:398] Leaking SignalData structure 0x7b080020a9a0 after lost signal to thread 8757
I20260709 13:03:31.816916 13643 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:31.818027  8605 debug-util.cc:398] Leaking SignalData structure 0x7b080018d040 after lost signal to thread 9355
W20260709 13:03:31.818372  8605 debug-util.cc:398] Leaking SignalData structure 0x7b08000c04e0 after lost signal to thread 9356
I20260709 13:03:31.819635  8656 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:31.809497  8579 rpcz_store.cc:276] 0709 13:03:30.564442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.314342 (+749900us) service_pool.cc:224] Handling call
0709 13:03:31.314669 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.767551 (+452882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.767721 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.768481  8725 rpcz_store.cc:269] 0709 13:03:29.110823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.408033 (+297210us) service_pool.cc:224] Handling call
0709 13:03:31.331806 (+1923773us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.765588  8699 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:40420 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2660)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2414)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2417)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2668)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2649)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2650)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2652)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2653)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2656)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2670)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2675)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2412)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2421)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2423)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2426)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2429)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2431)
W20260709 13:03:31.822654  8622 connection.cc:256] RPC call timeout handler was delayed by -1.0504917480000131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
W20260709 13:03:31.808679  8726 rpcz_store.cc:269] 0709 13:03:29.391098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.394331 (+  3233us) service_pool.cc:224] Handling call
0709 13:03:31.767834 (+2373503us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.808076  8577 rpcz_store.cc:276] 0709 13:03:30.564083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.260698 (+696615us) service_pool.cc:224] Handling call
0709 13:03:31.807568 (+546870us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.829205 12171 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:31.804981  8477 catalog_manager.cc:5697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:31.833688 13559 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.834496 13617 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:31.835991 13559 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:31.836629  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2678) took 2506 ms (client timeout 1934 ms). Trace:
W20260709 13:03:31.836943  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2660) took 3026 ms (client timeout 2591 ms). Trace:
I20260709 13:03:31.837042  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:31.837069  8733 rpcz_store.cc:269] 0709 13:03:29.319915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.823014 (+2503099us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.826661 (+  3647us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.837494  8727 rpcz_store.cc:269] 0709 13:03:28.810820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.836684 (+3025864us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.836865 (+   181us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.806703  9322 pending_rounds.cc:107] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 13:03:31.842679  9322 raft_consensus.cc:2889] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:31.827661  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2224) took 1275 ms. Trace:
I20260709 13:03:31.844030  8578 rpcz_store.cc:276] 0709 13:03:30.552054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.324236 (+772182us) service_pool.cc:224] Handling call
0709 13:03:31.827596 (+503360us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.828444  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 909) took 3081 ms (client timeout 1776 ms). Trace:
W20260709 13:03:31.847146  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2824) took 2529 ms (client timeout 1738 ms). Trace:
W20260709 13:03:31.848008  8575 rpcz_store.cc:269] 0709 13:03:29.318093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.140950 (+822857us) service_pool.cc:224] Handling call
0709 13:03:31.846883 (+1705933us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003a4154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:31.847060 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3098042240}
I20260709 13:03:31.768781 13568 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:31.771759 13629 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.855559  8650 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:31.846954  8728 rpcz_store.cc:269] 0709 13:03:28.747353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.801747 (+ 54394us) service_pool.cc:224] Handling call
0709 13:03:31.776374 (+2974627us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b5400229f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:31.828371 (+ 51997us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6093854976}
I20260709 13:03:31.876556  8651 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:31.882071  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:31.797128 13640 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:31.885589  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2414) took 2493 ms (client timeout 2139 ms). Trace:
W20260709 13:03:31.886026  8727 rpcz_store.cc:269] 0709 13:03:29.363106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.846687 (+2483581us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.856597 (+  9910us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.886610  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:31.887019  8729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2671) took 2759 ms (client timeout 2216 ms). Trace:
W20260709 13:03:31.887276  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:31.887420  8729 rpcz_store.cc:269] 0709 13:03:29.097710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.842340 (+2744630us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.857253 (+ 14913us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.870244  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:31.888121  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2677) took 2540 ms (client timeout 1924 ms). Trace:
W20260709 13:03:31.881429  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 253758us
W20260709 13:03:31.888576  8730 rpcz_store.cc:269] 0709 13:03:29.317641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.819980 (+2502339us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.857360 (+ 37380us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.819432 13643 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:31.889930  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2680) took 2458 ms (client timeout 2026 ms). Trace:
W20260709 13:03:31.889961  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2417) took 2494 ms (client timeout 2183 ms). Trace:
W20260709 13:03:31.891003  8733 rpcz_store.cc:269] 0709 13:03:29.423794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.850979 (+2427185us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.882152 (+ 31173us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.891052  8724 rpcz_store.cc:269] 0709 13:03:29.391474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.850948 (+2459474us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.885431 (+ 34483us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.892050  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 914) took 1765 ms (client timeout 1748 ms). Trace:
W20260709 13:03:31.892442  8733 rpcz_store.cc:269] 0709 13:03:30.126392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.891881 (+1765489us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.891992 (+   111us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.837428  9325 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:31.895218  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 923) took 1761 ms (client timeout 1605 ms). Trace:
W20260709 13:03:31.895604  8731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 910) took 3146 ms (client timeout 2842 ms). Trace:
W20260709 13:03:31.895674  8728 rpcz_store.cc:269] 0709 13:03:30.133389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.890082 (+1756693us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.895131 (+  5049us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.896080  8731 rpcz_store.cc:269] 0709 13:03:28.748943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.882043 (+3133100us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.895536 (+ 13493us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 13:03:31.898608  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2681) took 2429 ms (client timeout 2334 ms). Trace:
W20260709 13:03:31.898998  8730 rpcz_store.cc:269] 0709 13:03:29.469458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.889254 (+2419796us) service_pool.cc:211] Skipping call since client already timed out
0709 13:03:31.898539 (+  9285us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:03:31.900779  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:31.901183  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:31.832650  8477 catalog_manager.cc:5697] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 10, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 10 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:31.902138  8725 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
W20260709 13:03:31.902772  8725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 919) took 1772 ms (client timeout 1911 ms). Trace:
I20260709 13:03:31.903106 13627 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.903879 13627 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.901674  8733 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:31.905072  9325 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:31.901301  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64401.3us
W20260709 13:03:31.903214  8725 rpcz_store.cc:269] 0709 13:03:30.129885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.900827 (+1770942us) service_pool.cc:224] Handling call
0709 13:03:31.902691 (+  1864us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.903445  8477 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 7, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 7 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:31.909698  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1139) took 3088 ms. Trace:
I20260709 13:03:31.903905  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:31.914347  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:31.915916  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 918) took 1786 ms (client timeout 1824 ms). Trace:
W20260709 13:03:31.916371  8733 rpcz_store.cc:269] 0709 13:03:30.129576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.897806 (+1768230us) service_pool.cc:224] Handling call
0709 13:03:31.915836 (+ 18030us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.912103  8696 leader_election.cc:400] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:31.910164  9322 rpcz_store.cc:276] 0709 13:03:28.817911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.818133 (+   222us) service_pool.cc:224] Handling call
0709 13:03:28.818396 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.326836 (+1508440us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b5400319320. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:03:31.883987 (+1557151us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.905792 (+ 21805us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1094412544}
I20260709 13:03:31.917451  8696 leader_election.cc:403] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:31.889333 13629 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:31.919435  8650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 794) took 3063 ms (client timeout 1864 ms). Trace:
I20260709 13:03:31.921105  8724 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
W20260709 13:03:31.924418  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2420) took 2508 ms (client timeout 2646 ms). Trace:
W20260709 13:03:31.924729  8724 rpcz_store.cc:269] 0709 13:03:29.415804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.903544 (+2487740us) service_pool.cc:224] Handling call
0709 13:03:31.924371 (+ 20827us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.925911  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:31.928392  8550 connection.cc:256] RPC call timeout handler was delayed by -1.8175036250000858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
I20260709 13:03:31.902716 13595 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:31.853528 13630 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:31.931845  8657 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:31.896951 13631 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:31.930974  8728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2418) took 2530 ms (client timeout 2674 ms). Trace:
W20260709 13:03:31.923437  8650 rpcz_store.cc:269] 0709 13:03:28.855663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.330403 (+1474740us) service_pool.cc:224] Handling call
0709 13:03:31.919234 (+1588831us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b540018ecd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:31.919374 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1290378624}
W20260709 13:03:31.933452  8728 rpcz_store.cc:269] 0709 13:03:29.400149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.906581 (+2506432us) service_pool.cc:224] Handling call
0709 13:03:31.930909 (+ 24328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.810374 13639 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.929281  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:31.922984  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.011657  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:31.995270  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.023664  8730 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:31.995352  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:32.023452  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 220835us
I20260709 13:03:32.028187  8731 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:03:31.996606  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:31.824183 13633 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:31.856113 13617 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 13:03:32.029445  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:31.932523 13631 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.024683  8726 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:32.030608  8725 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:32.031847  8727 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
W20260709 13:03:32.037534  8725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 920) took 1907 ms (client timeout 2204 ms). Trace:
W20260709 13:03:32.035013  8550 connection.cc:256] RPC call timeout handler was delayed by -1.735603625000067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
W20260709 13:03:32.038112  8550 connection.cc:256] RPC call timeout handler was delayed by -1.7306836200000362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
W20260709 13:03:32.038038  8725 rpcz_store.cc:269] 0709 13:03:30.132290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.920938 (+1788648us) service_pool.cc:224] Handling call
0709 13:03:32.037475 (+116537us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.038672  8550 connection.cc:256] RPC call timeout handler was delayed by -1.4075411520000216s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2238s for a response
W20260709 13:03:32.036886  8726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 917) took 1909 ms (client timeout 2057 ms). Trace:
W20260709 13:03:32.037595  8727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2415) took 2661 ms (client timeout 2840 ms). Trace:
W20260709 13:03:32.039381  8727 rpcz_store.cc:269] 0709 13:03:29.376445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.910381 (+2533936us) service_pool.cc:224] Handling call
0709 13:03:32.037541 (+127160us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.040628 13546 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 13:03:32.040843  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.041455 13405 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:32.041878  8576 raft_consensus.cc:2410] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:03:32.042104 13405 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.043013  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.043644  8576 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:32.044404 13405 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.044561  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.045746  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.046980  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.048699  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.049278  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.050081  8583 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:32.050308  8579 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:32.052325 13627 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.053877  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.054258 13630 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.028837  8732 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:32.045234  8576 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:32.040518  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:32.057443  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 916) took 1906 ms (client timeout 1864 ms). Trace:
W20260709 13:03:32.058297  8726 rpcz_store.cc:269] 0709 13:03:30.127559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.908662 (+1781103us) service_pool.cc:224] Handling call
0709 13:03:32.036767 (+128105us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.057876  8730 rpcz_store.cc:269] 0709 13:03:30.127119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.899609 (+1772490us) service_pool.cc:224] Handling call
0709 13:03:32.033016 (+133407us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.054641  8584 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:32.061864  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2425) took 2602 ms (client timeout 2826 ms). Trace:
W20260709 13:03:32.062309  8732 rpcz_store.cc:269] 0709 13:03:29.452453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.911175 (+2458722us) service_pool.cc:224] Handling call
0709 13:03:32.055166 (+143991us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.061910  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 131831us
W20260709 13:03:32.063725  8731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2419) took 2655 ms (client timeout 2542 ms). Trace:
W20260709 13:03:32.064205  8731 rpcz_store.cc:269] 0709 13:03:29.407925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.896862 (+2488937us) service_pool.cc:224] Handling call
0709 13:03:32.063639 (+166777us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.063596  8550 connection.cc:256] RPC call timeout handler was delayed by -1.3495036250000112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25910000000000005s for a response
W20260709 13:03:32.065721  8550 connection.cc:256] RPC call timeout handler was delayed by -1.1599723389999781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1724s for a response
I20260709 13:03:32.067458  8724 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:32.068934  8724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 921) took 1936 ms (client timeout 2448 ms). Trace:
W20260709 13:03:32.069479  8724 rpcz_store.cc:269] 0709 13:03:30.132781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.935248 (+1802467us) service_pool.cc:224] Handling call
0709 13:03:32.068849 (+133601us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.930941 13585 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.098s
I20260709 13:03:32.072203  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.073518  8582 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:32.028481 13593 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:32.083062  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.089908  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.090724  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 791) took 3231 ms. Trace:
I20260709 13:03:32.093000  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:32.094275  8584 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:03:32.088655  8729 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:32.071060 13642 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:32.092984  8652 rpcz_store.cc:276] 0709 13:03:28.855180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.798933 (+2943753us) service_pool.cc:224] Handling call
0709 13:03:31.799310 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.799676 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:31.889734 (+ 90058us) log.cc:844] Serialized 14 byte log entry
0709 13:03:32.086108 (+196374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.086193 (+    85us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:32.086296 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.086389 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.086568 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":123286,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":1805,"raft.queue_time_us":5233,"thread_start_us":252794,"threads_started":2}
I20260709 13:03:32.097587  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:32.098109 13611 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.099615 13627 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:32.100041 13627 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:32.100392 13627 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:32.103013 13631 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.103363  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.105496  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.106441  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.108487  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2649) took 3339 ms. Trace:
I20260709 13:03:32.110936  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2650) took 3341 ms. Trace:
I20260709 13:03:32.111377  8732 rpcz_store.cc:276] 0709 13:03:28.769692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.110013 (+3340321us) service_pool.cc:224] Handling call
0709 13:03:32.110375 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.110601 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.110628 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.110686 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:32.110720 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.110776 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.110871 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.113538  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.114451  8549 leader_election.cc:336] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.599s (SENT)
W20260709 13:03:32.114943  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 215134us
I20260709 13:03:32.076692 13405 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.124679 13405 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.126668  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.127202 13405 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.127414  9322 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:32.130497  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.108860  8727 rpcz_store.cc:276] 0709 13:03:28.769378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.107589 (+3338211us) service_pool.cc:224] Handling call
0709 13:03:32.107941 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.108140 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.108168 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.108211 (+    43us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:32.108250 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.108304 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.108402 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.123267  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.123952  8726 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:32.124066  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2653) took 3353 ms. Trace:
I20260709 13:03:32.133141  8731 rpcz_store.cc:276] 0709 13:03:28.770506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.114746 (+3344240us) service_pool.cc:224] Handling call
0709 13:03:32.123457 (+  8711us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.123684 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.123709 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.123745 (+    36us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:32.123833 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.123885 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.123989 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.134250  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 922) took 2000 ms. Trace:
I20260709 13:03:32.134694  8726 rpcz_store.cc:276] 0709 13:03:30.133076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.059222 (+1926146us) service_pool.cc:224] Handling call
0709 13:03:32.132967 (+ 73745us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.137759  8732 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:32.137883  8730 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:32.139156  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2668) took 3111 ms. Trace:
I20260709 13:03:32.139467  8730 rpcz_store.cc:276] 0709 13:03:29.027389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.072003 (+3044614us) service_pool.cc:224] Handling call
0709 13:03:32.139099 (+ 67096us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.131191  8724 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:32.139606  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:32.145632  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70211.4us
W20260709 13:03:32.147822  9296 connection.cc:256] RPC call timeout handler was delayed by -2.3713614749999579s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
I20260709 13:03:32.148533  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2652) took 3372 ms. Trace:
I20260709 13:03:32.148941  8732 rpcz_store.cc:276] 0709 13:03:28.770239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.112543 (+3342304us) service_pool.cc:224] Handling call
0709 13:03:32.112947 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.138231 (+ 25284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.142681 (+  4450us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.152763  9325 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:32.117650 13631 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:03:32.154793  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08002e4f20 after lost signal to thread 7484
W20260709 13:03:32.163125  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080030f740 after lost signal to thread 8516
W20260709 13:03:32.164675  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080030f4c0 after lost signal to thread 8608
W20260709 13:03:32.165092  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080030f3e0 after lost signal to thread 8609
W20260709 13:03:32.155236 13348 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 13:03:32.158412  8726 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
W20260709 13:03:32.167263  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800189920 after lost signal to thread 8650
W20260709 13:03:32.167716  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001caec0 after lost signal to thread 8651
I20260709 13:03:32.168936  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 797) took 3312 ms. Trace:
W20260709 13:03:32.173053  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001cafa0 after lost signal to thread 8652
W20260709 13:03:32.174667  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001c3960 after lost signal to thread 8653
W20260709 13:03:32.175202  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800189840 after lost signal to thread 8654
W20260709 13:03:32.175472  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001cafc0 after lost signal to thread 8655
I20260709 13:03:32.178927  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2669) took 3127 ms. Trace:
I20260709 13:03:32.179363  8726 rpcz_store.cc:276] 0709 13:03:29.044890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.141008 (+3096118us) service_pool.cc:224] Handling call
0709 13:03:32.158205 (+ 17197us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:32.171856 (+ 13651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.172077 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.176160  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001ee0c0 after lost signal to thread 8656
W20260709 13:03:32.302062  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001ee0a0 after lost signal to thread 8657
W20260709 13:03:32.302246 13648 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.248s	user 0.000s	sys 0.005s
W20260709 13:03:32.302527  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800107ce0 after lost signal to thread 8659
I20260709 13:03:32.193307  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:32.303187  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080010c900 after lost signal to thread 8683
I20260709 13:03:32.302894 13630 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 13:03:32.303567  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080010c7e0 after lost signal to thread 8696
I20260709 13:03:32.303689  9326 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:32.303879  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080010c780 after lost signal to thread 8698
W20260709 13:03:32.304163  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080010c7c0 after lost signal to thread 8699
W20260709 13:03:32.304764  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08002fb8a0 after lost signal to thread 8725
W20260709 13:03:32.305044  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080021ea60 after lost signal to thread 8726
I20260709 13:03:32.304908  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.193046  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:32.305851  9326 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:32.306335 13595 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:32.306819 13595 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:32.307000  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.306583 13568 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.307255 13595 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:32.308234 13568 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:32.308490 13653 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.274s	user 0.001s	sys 0.002s
W20260709 13:03:32.310029  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 117128us
I20260709 13:03:32.216923 13640 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.313520 13640 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:32.314105  9297 connection.cc:256] RPC call timeout handler was delayed by -2.7064940059999572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2416s for a response
W20260709 13:03:32.194159  7486 debug-util.cc:398] Leaking SignalData structure 0x7b080000fde0 after lost signal to thread 8622
I20260709 13:03:32.315958 13624 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:32.316569 13624 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.317008  9326 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:32.318020  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.318583  9326 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:32.318895 13624 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:32.319763 13624 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:32.320362 13624 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.198995  8724 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 13:03:32.321810  8724 pending_rounds.cc:85] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 4
I20260709 13:03:32.322201  8724 pending_rounds.cc:107] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 13:03:32.322562  8724 raft_consensus.cc:2889] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:32.323582  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2675) took 3212 ms. Trace:
W20260709 13:03:32.214450  9296 connection.cc:256] RPC call timeout handler was delayed by -1.5520729909999318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
W20260709 13:03:32.324582  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.324852  9297 connection.cc:256] RPC call timeout handler was delayed by -2.1300779569999122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2842s for a response
W20260709 13:03:32.325699  9297 connection.cc:256] RPC call timeout handler was delayed by -1.8566630809999651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1748s for a response
I20260709 13:03:32.178038 13586 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.326354  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:32.324939  9296 connection.cc:256] RPC call timeout handler was delayed by -1.4044729909999205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20260709 13:03:32.327284  9296 connection.cc:256] RPC call timeout handler was delayed by -1.3270729909999091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2143s for a response
I20260709 13:03:32.219311 13633 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:32.327679  9296 connection.cc:256] RPC call timeout handler was delayed by -1.2488681299998916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260709 13:03:32.182001  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2656) took 3407 ms. Trace:
I20260709 13:03:32.328212  8725 rpcz_store.cc:276] 0709 13:03:28.771296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.115610 (+3344314us) service_pool.cc:224] Handling call
0709 13:03:32.152831 (+ 37221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.171305 (+ 18474us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:32.171336 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.178470 (+  7134us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:32.178513 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.178576 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.178699 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20732,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3985}
I20260709 13:03:32.323623 13624 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.029811  8477 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 8, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 8 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:32.237630  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.331286  9328 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:32.332216  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.332866  9328 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:32.218847  8732 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:32.332290  8477 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 8, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 8 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:32.334132  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2676) took 3057 ms. Trace:
I20260709 13:03:32.188726  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.201771  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1137) took 3385 ms. Trace:
I20260709 13:03:32.334607  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2657) took 3554 ms. Trace:
I20260709 13:03:32.237412  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:32.294075  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 116891us
I20260709 13:03:32.325733  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:32.233706  8729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 915) took 2101 ms (client timeout 1785 ms). Trace:
I20260709 13:03:32.339795  8727 rpcz_store.cc:276] 0709 13:03:28.779799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.141002 (+3361203us) service_pool.cc:224] Handling call
0709 13:03:32.228097 (+ 87095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.228342 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.228397 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.228460 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:32.228517 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.228593 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.334414 (+105821us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b5400088ed0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:32.334544 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100323712}
I20260709 13:03:32.330145  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:32.340546  8729 rpcz_store.cc:269] 0709 13:03:30.126799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.894819 (+1768020us) service_pool.cc:224] Handling call
0709 13:03:32.227741 (+332922us) spinlock_profiling.cc:243] Waited 64.3 ms on lock 0x7b5400088ed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:32.227950 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":147874176}
I20260709 13:03:32.243382  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2611) took 2910 ms. Trace:
I20260709 13:03:32.338434  8732 rpcz_store.cc:276] 0709 13:03:29.276973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.172400 (+2895427us) service_pool.cc:224] Handling call
0709 13:03:32.218405 (+ 46005us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.333819 (+115414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.334078 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.324373  8584 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:32.244671  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.247114  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2670) took 3157 ms. Trace:
I20260709 13:03:32.340636  9325 rpcz_store.cc:276] 0709 13:03:28.816204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.816339 (+   135us) service_pool.cc:224] Handling call
0709 13:03:28.816653 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.202766 (+1386113us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b54004de520. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:03:32.153188 (+1950422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.201685 (+ 48497us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78515072}
I20260709 13:03:32.276319  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:32.324079  8724 rpcz_store.cc:276] 0709 13:03:29.111180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.146387 (+3035207us) service_pool.cc:224] Handling call
0709 13:03:32.185201 (+ 38814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.323317 (+138116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.323500 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.342645  8657 rpcz_store.cc:276] 0709 13:03:29.333131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.813229 (+2480098us) service_pool.cc:224] Handling call
0709 13:03:31.813633 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.932266 (+118633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.243008 (+310742us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540013a954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:32.243288 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271060736}
I20260709 13:03:32.344470  8730 rpcz_store.cc:276] 0709 13:03:29.075956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.140560 (+3064604us) service_pool.cc:224] Handling call
0709 13:03:32.183944 (+ 43384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.227963 (+ 44019us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.228002 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.232991 (+  4989us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:32.233097 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.233187 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.233343 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.326411  9297 connection.cc:256] RPC call timeout handler was delayed by -1.823363081000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
W20260709 13:03:32.346638  9297 connection.cc:256] RPC call timeout handler was delayed by -1.7522630810000237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20260709 13:03:32.347103  9297 connection.cc:256] RPC call timeout handler was delayed by -1.5785630809999702s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
I20260709 13:03:32.347184  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:32.347184  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:32.263537  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.236800  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.216900  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:32.345074  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82267.2us
I20260709 13:03:32.348990  9330 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:32.153177 13559 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:32.305384  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08002e56a0 after lost signal to thread 8729
I20260709 13:03:32.173341  8658 rpcz_store.cc:276] 0709 13:03:28.856167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.800087 (+2943920us) service_pool.cc:224] Handling call
0709 13:03:31.800603 (+   516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.800869 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:31.994882 (+194013us) log.cc:844] Serialized 14 byte log entry
0709 13:03:32.168440 (+173558us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.168516 (+    76us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:32.168591 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.168694 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.168857 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":227883,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":18223,"thread_start_us":199209,"threads_started":2}
I20260709 13:03:32.352294  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:32.195849  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:32.355644  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08002fb960 after lost signal to thread 8730
W20260709 13:03:32.355990  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001e6a40 after lost signal to thread 8732
W20260709 13:03:32.350260  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.348969  9297 connection.cc:256] RPC call timeout handler was delayed by -1.4171356419999483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1605s for a response
I20260709 13:03:32.354570  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:32.357101  9297 connection.cc:256] RPC call timeout handler was delayed by -1.220035641999857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
W20260709 13:03:32.360960  9297 connection.cc:256] RPC call timeout handler was delayed by -1.1893941429999586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4039s for a response
W20260709 13:03:32.356611  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080014dc40 after lost signal to thread 8757
W20260709 13:03:32.363006  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003e3900 after lost signal to thread 9295
W20260709 13:03:32.363278  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003c04c0 after lost signal to thread 9296
I20260709 13:03:32.366082  8733 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:32.370313  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.361960  9297 connection.cc:256] RPC call timeout handler was delayed by -1.1417356419998441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
W20260709 13:03:32.378192  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce980 after lost signal to thread 9297
I20260709 13:03:32.380983 13586 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.381762 13586 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.383467 13629 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.383893 13595 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.384171 13586 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.388952  9323 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:32.350633 13633 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:32.385144  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800338aa0 after lost signal to thread 9322
I20260709 13:03:32.395596  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:32.400018  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800106640 after lost signal to thread 9325
W20260709 13:03:32.400810  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080037d220 after lost signal to thread 9331
W20260709 13:03:32.401690  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003e77c0 after lost signal to thread 9355
W20260709 13:03:32.402187  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080028b360 after lost signal to thread 9356
W20260709 13:03:32.402593  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003ed420 after lost signal to thread 12171
I20260709 13:03:32.287904 13593 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 13:03:32.409577  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.404475  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.355003 13559 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:32.410786  8728 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:32.398170 13428 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:32.412942  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:32.413524  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.413947  8732 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
W20260709 13:03:32.415575  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 184259us
I20260709 13:03:32.415809  8477 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 9, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 9 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.416826  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.417631 13640 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:32.408550  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800392020 after lost signal to thread 13125
I20260709 13:03:32.263273 13585 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.128s
W20260709 13:03:32.419279  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800367260 after lost signal to thread 13156
W20260709 13:03:32.431458  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc7e0 after lost signal to thread 13158
W20260709 13:03:32.431869  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080037ff80 after lost signal to thread 13183
I20260709 13:03:32.427321  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:32.434226  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003eb020 after lost signal to thread 13236
W20260709 13:03:32.434731  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001b5400 after lost signal to thread 13247
W20260709 13:03:32.435150  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003d1c20 after lost signal to thread 13248
W20260709 13:03:32.435551  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003d8440 after lost signal to thread 13302
W20260709 13:03:32.435865  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003e1720 after lost signal to thread 13328
W20260709 13:03:32.436200  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800381e60 after lost signal to thread 13340
W20260709 13:03:32.436489  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080039ae40 after lost signal to thread 13341
W20260709 13:03:32.436856  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080025aec0 after lost signal to thread 13368
W20260709 13:03:32.437170  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003bad60 after lost signal to thread 13387
I20260709 13:03:32.418066  8732 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:32.441987  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800109e00 after lost signal to thread 13390
I20260709 13:03:32.416682  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:32.406257 13633 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:32.390211 13400 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:32.413133 13428 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:32.434666  8733 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
W20260709 13:03:32.442394  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080027a5c0 after lost signal to thread 13397
W20260709 13:03:32.460644  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080027fc00 after lost signal to thread 13400
W20260709 13:03:32.460986  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003882e0 after lost signal to thread 13405
W20260709 13:03:32.461215  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080034bd40 after lost signal to thread 13417
W20260709 13:03:32.461505  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800300b40 after lost signal to thread 13428
W20260709 13:03:32.461764  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003e1220 after lost signal to thread 13429
W20260709 13:03:32.462005  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800284480 after lost signal to thread 13445
W20260709 13:03:32.462270  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080039ff40 after lost signal to thread 13446
W20260709 13:03:32.462554  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800238ba0 after lost signal to thread 13466
W20260709 13:03:32.462831  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080034a8c0 after lost signal to thread 13467
W20260709 13:03:32.463100  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800092ba0 after lost signal to thread 13471
W20260709 13:03:32.439713  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44371.8us
I20260709 13:03:32.443425  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.451256  9328 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:32.455835  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2666) took 3579 ms. Trace:
I20260709 13:03:32.458024  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.419713 13559 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.465261  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:32.467707  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 59509.6us
I20260709 13:03:32.470147  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.471518  8725 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:32.472537  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003bf560 after lost signal to thread 13478
W20260709 13:03:32.472939  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003ab4e0 after lost signal to thread 13485
W20260709 13:03:32.473196  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080027e6c0 after lost signal to thread 13491
W20260709 13:03:32.473510  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800361f60 after lost signal to thread 13497
I20260709 13:03:32.464321  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.483349  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2412) took 3157 ms. Trace:
I20260709 13:03:32.484053  8726 rpcz_store.cc:276] 0709 13:03:29.325888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.481858 (+3155970us) service_pool.cc:224] Handling call
0709 13:03:32.482387 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.482769 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.482810 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.482897 (+    87us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:32.482955 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.483045 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.483214 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:32.489959  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003d9340 after lost signal to thread 13516
W20260709 13:03:32.490566  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800137320 after lost signal to thread 13559
W20260709 13:03:32.490888  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003d5420 after lost signal to thread 13569
W20260709 13:03:32.491180  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800313da0 after lost signal to thread 13576
W20260709 13:03:32.491485  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080035dec0 after lost signal to thread 13580
W20260709 13:03:32.491791  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003581a0 after lost signal to thread 13585
W20260709 13:03:32.492069  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003ee960 after lost signal to thread 13586
W20260709 13:03:32.492369  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003567a0 after lost signal to thread 13593
W20260709 13:03:32.492653  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08002f1ea0 after lost signal to thread 13595
W20260709 13:03:32.492990  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800312fa0 after lost signal to thread 13598
I20260709 13:03:32.494628  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:32.493280  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080036d120 after lost signal to thread 13599
W20260709 13:03:32.497033  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800338260 after lost signal to thread 13603
I20260709 13:03:32.455886 13633 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:32.502177  8730 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:32.462647 13428 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:32.506815  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:32.506757  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.508142  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:32.510831  8727 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:32.472786 13405 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:32.463831  8731 rpcz_store.cc:276] 0709 13:03:28.870283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.134095 (+3263812us) service_pool.cc:224] Handling call
0709 13:03:32.156453 (+ 22358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.156690 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:32.157045 (+   355us) log.cc:844] Serialized 14 byte log entry
0709 13:03:32.437997 (+280952us) log.cc:844] Serialized 14 byte log entry
0709 13:03:32.438370 (+   373us) log.cc:844] Serialized 14 byte log entry
0709 13:03:32.449630 (+ 11260us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.449693 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:32.449764 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.449853 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.449995 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":926,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":2795,"raft.queue_time_us":80945,"thread_start_us":252257,"threads_started":1}
I20260709 13:03:32.508849  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:32.532447  8733 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:32.541682  9328 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:32.540097  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.543193  9327 raft_consensus.cc:2410] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:03:32.546396  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.496009 13631 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 13:03:32.418051  8477 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 7, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 7 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.535017 13405 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:32.555377  8727 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:32.555863  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.548094  9297 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:32.562124  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2427) took 3081 ms. Trace:
I20260709 13:03:32.563164  8724 rpcz_store.cc:276] 0709 13:03:29.480459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.517886 (+3037427us) service_pool.cc:224] Handling call
0709 13:03:32.561486 (+ 43600us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.561689 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.561727 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.561785 (+    58us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:32.561835 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.561903 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.562048 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.562638  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2426) took 3094 ms. Trace:
W20260709 13:03:32.562201  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55508.8us
I20260709 13:03:32.565698  8727 rpcz_store.cc:276] 0709 13:03:29.468080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.516812 (+3048732us) service_pool.cc:224] Handling call
0709 13:03:32.552907 (+ 36095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.562407 (+  9500us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.562586 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.542507 13400 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:32.567780  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2421) took 3136 ms. Trace:
I20260709 13:03:32.568202  8728 rpcz_store.cc:276] 0709 13:03:29.431329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.503719 (+3072390us) service_pool.cc:224] Handling call
0709 13:03:32.550742 (+ 47023us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.555100 (+  4358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.555126 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.567493 (+ 12367us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:32.567538 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.567603 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.567720 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.545812 13629 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.578176  8732 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:32.579656  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2428) took 3081 ms. Trace:
I20260709 13:03:32.580071  8732 rpcz_store.cc:276] 0709 13:03:29.498510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.526140 (+3027630us) service_pool.cc:224] Handling call
0709 13:03:32.569498 (+ 43358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.579301 (+  9803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.579581 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.533778 13593 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 13:03:32.581863  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.585870  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2429) took 3085 ms. Trace:
I20260709 13:03:32.586330  8731 rpcz_store.cc:276] 0709 13:03:29.500255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.532634 (+3032379us) service_pool.cc:224] Handling call
0709 13:03:32.573971 (+ 41337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.585486 (+ 11515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.585515 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.585571 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:32.585612 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.585670 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.585799 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.565958 13405 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.597536  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.594767  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2430) took 2632 ms. Trace:
I20260709 13:03:32.598181  8724 rpcz_store.cc:276] 0709 13:03:29.962489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.565818 (+2603329us) service_pool.cc:224] Handling call
0709 13:03:32.583147 (+ 17329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.583332 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.583357 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.594469 (+ 11112us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:32.594522 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.594585 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.594706 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.604597  8727 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:32.605335  8728 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:32.607645  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.608076  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:32.608489  8658 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:32.609673 13617 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:32.610116 13617 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:32.611078 13617 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.612270 13617 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:32.609673  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.574467 13624 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:32.589072 13242 raft_consensus.cc:2764] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
W20260709 13:03:32.616755 13661 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 13:03:32.617385 13658 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:03:32.609997  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:32.628430  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.596504 13595 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.604882 13593 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:32.633072  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2683) took 1377 ms. Trace:
I20260709 13:03:32.633548  8727 rpcz_store.cc:276] 0709 13:03:31.255684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.575980 (+1320296us) service_pool.cc:224] Handling call
0709 13:03:32.600547 (+ 24567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.605755 (+  5208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.633001 (+ 27246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.634421  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2431) took 1227 ms. Trace:
I20260709 13:03:32.636602  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.634778  8728 rpcz_store.cc:276] 0709 13:03:31.407064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.577287 (+1170223us) service_pool.cc:224] Handling call
0709 13:03:32.605161 (+ 27874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.634189 (+ 29028us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.634370 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:32.640376  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:32.634019 13611 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.523s	user 0.007s	sys 0.004s
I20260709 13:03:32.656031  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.659198  9326 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:32.661862  8650 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:32.662303  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.666582  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.668316  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.671082  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.673520  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.674026  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:32.660392 13656 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.312s	user 0.001s	sys 0.000s
I20260709 13:03:32.674652  8654 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:32.674652  8655 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:32.678752  8733 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:32.679644  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.682626  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:32.682826  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 152541us
W20260709 13:03:32.685034  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 144370us
I20260709 13:03:32.689072  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.692619  8653 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
W20260709 13:03:32.692955 13663 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:03:32.713878  8477 catalog_manager.cc:5697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.689996  9326 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:03:32.729465 13668 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 13:03:32.735903 13627 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.597s	user 0.011s	sys 0.000s
W20260709 13:03:32.738528 13659 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 13:03:32.716288  8477 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:32.750236  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:32.755251  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 88678us
W20260709 13:03:32.756093  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.757364  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 210029us
I20260709 13:03:32.758293  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:32.760547 13568 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:32.760972 13568 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:32.761341 13568 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.736557 13627 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.765617  8576 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:32.766783 13627 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.764627 13428 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.768883 13568 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.771221 13642 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:32.771554  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.771838 13617 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:32.775326  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.771695 13642 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:32.751415  8550 leader_election.cc:336] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Remote error: Timed out: Call waited in the queue past client deadline
W20260709 13:03:32.781617  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.775943  8652 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:32.787853  8659 raft_consensus.cc:2368] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11.
W20260709 13:03:32.788194 13665 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 13:03:32.788980  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.788997  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.790280  8653 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:03:32.785987  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2422) took 3354 ms. Trace:
I20260709 13:03:32.792006  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.792783  9325 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:32.793884  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.780442  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.797801  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.798461  9328 raft_consensus.cc:2376] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 13. Re-sending same reply.
W20260709 13:03:32.801549  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.799697 13328 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:32.803675 13328 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:32.804459 13328 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.805754 13328 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.806210  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.776065 13639 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:32.808281  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.810039 13568 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.814649 13428 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.816378  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.818214  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:32.818332  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 181792us
I20260709 13:03:32.819028  9322 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:03:32.819710 13242 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:32.820588  9322 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:03:32.820389 13242 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:32.807825 13642 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.824126 13642 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:32.825786  8477 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 8, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 8 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.827170  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.828576  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.828038  8477 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 6 to 9, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 9 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:32.830271  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.829722  8477 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 9 to 10, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 10 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:32.866297  8477 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:03:32.832509  9297 leader_election.cc:336] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Remote error: Timed out: Call waited in the queue past client deadline
I20260709 13:03:32.833428  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.867583  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:32.837481  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 95567.7us
I20260709 13:03:32.792070  8729 rpcz_store.cc:276] 0709 13:03:29.431670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.504072 (+3072402us) service_pool.cc:224] Handling call
0709 13:03:32.531306 (+ 27234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.575032 (+ 43726us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:32.632445 (+ 57413us) log.cc:844] Serialized 14 byte log entry
0709 13:03:32.785663 (+153218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:32.785715 (+    52us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:32.785758 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.785820 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:32.785933 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126,"prepare.queue_time_us":29991,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":7167,"thread_start_us":145578,"threads_started":1}
I20260709 13:03:32.867930  8477 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 6, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 6 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.838047 13627 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:32.869585  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.870720  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.871434  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:32.871963 13348 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:32.872921  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.854826  8653 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 13:03:32.856348  8659 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:03:32.879913  8653 pending_rounds.cc:85] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 7
I20260709 13:03:32.879989  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.880426  8653 pending_rounds.cc:107] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 13:03:32.858785 13633 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.879324  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:32.881431  8659 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.881979  8657 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:32.883133 13559 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.883962  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.884562  8657 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:32.859588  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:32.885711  8652 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:32.862897  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:32.864073  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.886063  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:32.887581  8655 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:32.864437 13568 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.864687  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.864835  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.888662 13559 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.888787 13676 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:32.889201  8658 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:32.889582 13676 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:32.873142 13348 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:32.890089 13676 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.889763 13568 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.890304 13348 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:32.891227  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:32.892153 13568 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.880854  8653 raft_consensus.cc:2889] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:32.893024  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:32.879351 13405 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:32.898813 13405 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.901620  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:32.903308  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.904050  8659 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:32.904594  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.905670  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.906052  9297 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:32.906292  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.907367 13622 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:32.907509  8655 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:03:32.907814 13622 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:32.908299 13622 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:32.908573 13624 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:32.908623  9297 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:32.909140 13624 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:32.909587 13624 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:32.914434  8650 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:32.914819  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.903358  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.916524  8651 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:32.920280 13586 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:32.920789 13586 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:32.921167 13586 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:32.921303  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:32.922931  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:32.926702  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:32.927601 13617 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:32.927982 13617 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:32.928300 13617 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.875101  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.875697  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.929786  9328 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:32.930828  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.876273  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.931525  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:32.932489  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.932643 13125 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:32.933012 13125 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:32.933374 13125 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:32.933604  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.934310  9323 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:32.935055  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.935645  9323 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:32.936432  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.876811  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.931540  9328 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:32.839777  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.877349  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.939904  9331 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:32.877724  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.858031  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:32.887400  8477 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 10, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 10 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.868428 13546 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:32.872594 13629 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:32.869529  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:32.931077 13624 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.928968 13586 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.850514  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.929715 13559 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:32.936645 13617 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.940666 13568 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
W20260709 13:03:32.953207  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.941763  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 72366.4us
W20260709 13:03:32.957083  8550 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Remote error: Timed out: Call waited in the queue past client deadline
I20260709 13:03:32.941936 13405 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:32.941066 13125 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.958138 13633 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.957700  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.960814  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.961256 13400 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:32.958199 13405 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:32.963263  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.963452 13405 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.964672  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.965611  9322 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:03:32.940418 13676 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.968551 13676 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:32.968634  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:32.943404  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:32.943580 13586 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:32.945923  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:32.970345  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:32.970580  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.971174 13546 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:32.942971 13640 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:32.971104  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:32.974265  8550 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Remote error: Timed out: Call waited in the queue past client deadline
I20260709 13:03:32.979199 13568 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:32.979980 13568 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:32.981040 13639 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:32.981442 13639 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:32.982079  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:32.982311 13639 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:32.982906 13400 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:32.983989 13639 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:32.970072  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.987087 13568 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:32.988258 13642 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:32.987145  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:32.988873 13642 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:32.989338 13568 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:32.989845 13568 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:32.990376 13627 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:32.990108 13642 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:32.947582  8653 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 13:03:32.943511 13624 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.973888 13125 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:32.974238 13676 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:32.946676 13617 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:32.947726  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:32.948628  9325 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:03:32.969694 13629 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:32.941789  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:32.946297  8654 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:32.991719 13642 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:32.972458  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:32.997730  9324 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:33.006489  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:33.024303  8650 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:33.014514  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:33.025305  9323 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:32.992093 13568 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:33.016126  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:33.018374  8697 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.576s (SENT)
I20260709 13:03:33.030122  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:33.019407  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:33.019467 13640 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:33.020231  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:33.027341  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 85621.4us
I20260709 13:03:33.032665  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:32.942379  8477 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 7 to 8, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 8 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:33.031885  8652 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
W20260709 13:03:33.035496  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 63085.4us
I20260709 13:03:33.036375  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:33.024259 13629 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:33.038028  9327 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:32.953310 13622 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.025058 13676 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:33.041631 13627 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:33.041770 13405 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:33.042356 13638 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:33.046707 13400 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:33.049070  8650 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:33.041133 13624 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:33.055308  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:33.067288 13624 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.067711 13624 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.067822  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:33.068799 13617 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:33.069180 13617 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.069555 13617 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:33.069744 13642 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:33.070318 13642 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:33.071255 13642 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.055918  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:32.957224 13348 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.074023 13640 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:33.074666  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:33.075445 13348 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:33.075624 13640 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:33.076135 13640 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.076514 13640 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:33.035450 13125 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:33.063330  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:33.064898  8656 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
W20260709 13:03:33.065357  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 246038us
W20260709 13:03:33.082944  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 114703us
I20260709 13:03:33.031275 13516 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 13:03:33.065191 13629 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:33.094698 13516 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:33.095075 13516 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:33.095975 13516 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.097688  8477 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 8 to 9, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 9 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:33.099601  8477 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 7, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 7 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:33.090595  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:33.090453 13405 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:33.064842 13683 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:33.093520 13633 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:33.093730 13622 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:33.102898  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 84561.1us
W20260709 13:03:33.107358  8697 leader_election.cc:336] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.694s (SENT)
W20260709 13:03:33.107708  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44442.4us
I20260709 13:03:33.108210  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:33.108850  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:33.088049 13586 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:33.121529  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:03:33.122632  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44811.8us
I20260709 13:03:33.110069 13497 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:33.123508 13497 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:33.123551  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:33.102978 13683 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:33.124398 13683 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.124521 13497 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.117161 13676 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:33.118718  9296 leader_election.cc:336] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Remote error: Timed out: Call waited in the queue past client deadline
I20260709 13:03:33.129743  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:33.121464 13400 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 13:03:33.137602 13400 consensus_queue.cc:1243] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:33.132180  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:33.125715 13633 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:33.116375  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2423) took 3684 ms. Trace:
I20260709 13:03:33.097182 13516 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:33.128388  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:33.133494 13125 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:33.135013  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:33.135553 13546 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:33.135623 13586 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.136219 13340 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:33.145277 12171 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:33.146765  8697 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.708s (SENT)
I20260709 13:03:33.147357  9324 pending_rounds.cc:85] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 9
I20260709 13:03:33.153364  9324 pending_rounds.cc:107] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:03:33.153882  9324 raft_consensus.cc:2889] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:33.145998  8625 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:33.147986  9322 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 8. Current term is 9. Ops: []
W20260709 13:03:33.154759  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.145474  8725 rpcz_store.cc:276] 0709 13:03:29.432003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.509598 (+3077595us) service_pool.cc:224] Handling call
0709 13:03:32.822502 (+312904us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:32.822691 (+   189us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:32.823074 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:32.823102 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:32.823742 (+   640us) log.cc:844] Serialized 52 byte log entry
0709 13:03:33.025534 (+201792us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:33.060873 (+ 35339us) log.cc:844] Serialized 14 byte log entry
0709 13:03:33.061289 (+   416us) log.cc:844] Serialized 14 byte log entry
0709 13:03:33.061502 (+   213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:33.061570 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:33.116107 (+ 54537us) raft_consensus.cc:1685] finished
0709 13:03:33.116128 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:33.116280 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23688,"prepare.queue_time_us":35237,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":11671,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":23917,"spinlock_wait_cycles":4140416,"thread_start_us":201416,"threads_started":1,"wal-append.queue_time_us":254751}
I20260709 13:03:33.153286  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:33.157269  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:33.159451  9326 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:33.160449  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:33.161001  8576 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:33.161381  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:33.162009  8575 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:33.164129  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:33.164804  8580 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:03:33.165760  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.166566  8579 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:33.145252 13633 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.137099 13629 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.183272 13328 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:33.185127 13624 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.197511  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.186035  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.186182  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:33.198155  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 69838.2us
I20260709 13:03:33.186753  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:33.186884  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.198931  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:33.199164  8726 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:33.187242  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.200282  8727 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:33.187554  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.200738  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:33.201118  8733 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:33.202272  8731 raft_consensus.cc:2410] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:33.203142  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:33.203456  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.151153 13559 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:33.151539 13638 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:33.191392  9330 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:33.194301  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:33.194669  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48757.5us
I20260709 13:03:33.143642 13640 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:33.214239  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.214146  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.209867  8730 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:33.155068 13586 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:33.197851  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:33.215494  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:33.195596 12171 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:33.182583 13622 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:33.206800  8575 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:33.207613  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:33.210449  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:33.214990  8724 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:33.204854  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.190804 13428 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:33.192731 13633 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:33.224175  8575 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:33.227118  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.228565  8578 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:33.228906  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:33.230294  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:33.207914 13585 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:33.205504 13599 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:33.233628  8732 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:33.247589  8580 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:33.251602  9326 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:33.259675  8579 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 11. Ops: []
I20260709 13:03:33.250541  8730 pending_rounds.cc:85] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:03:33.260311  8730 pending_rounds.cc:107] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 13:03:33.260617  8730 raft_consensus.cc:2889] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:33.219087 13559 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:33.159096  9327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:33.264667  9327 raft_consensus.cc:606] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:33.265168  9327 raft_consensus.cc:614] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:33.274030  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.277046  8580 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:33.264562  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:33.278088  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 143158us
I20260709 13:03:33.223042 13638 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:33.285301 13585 raft_consensus.cc:2764] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:03:33.291399  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.292783  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.293655  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.294826  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:33.304464  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 83483.1us
I20260709 13:03:33.306867  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.307816  8580 raft_consensus.cc:2410] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 13:03:33.321825  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 111428us
I20260709 13:03:33.242890 13568 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.325820  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.347419  8732 pending_rounds.cc:85] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:03:33.347982  8732 pending_rounds.cc:107] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:33.348305  8732 raft_consensus.cc:2889] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:33.164876  8477 catalog_manager.cc:5697] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 6, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 6 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:03:33.358385  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:33.399287 13689 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
W20260709 13:03:33.535679  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.363022 13622 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.616724  8581 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:33.320639 13585 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260709 13:03:33.578166  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:33.585824  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.587630  8576 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:33.627768  8580 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:33.588251  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.619882  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:33.628634  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:33.621798  8549 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.216s (SENT)
I20260709 13:03:33.630760  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:33.632943  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 279408us
I20260709 13:03:33.636938  8579 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:33.633843  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:33.655726  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 314706us
I20260709 13:03:33.655632  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:33.587073 13429 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:33.657051  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:03:33.661774  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.662214  8724 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:03:33.616916 13622 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:33.668340  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 337424us
I20260709 13:03:33.415203 13559 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:33.668385  9295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.000s	sys 0.023s
I20260709 13:03:33.648995 13428 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:33.552163 13638 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.574241 13694 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.100s
I20260709 13:03:33.632009 13639 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
W20260709 13:03:33.737635  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 411958us
I20260709 13:03:33.676698 13642 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:33.742703 13629 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:33.743880  9331 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:33.743081  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.744513  9331 raft_consensus.cc:606] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:33.744894  9331 raft_consensus.cc:614] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:33.728595 13599 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:33.741606  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:33.752960  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 119220us
I20260709 13:03:33.754586  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:33.756183  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:33.759922  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 103007us
I20260709 13:03:33.762383  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:33.676883 13429 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.767290  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:33.771878  8731 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.568s	user 0.009s	sys 0.001s
I20260709 13:03:33.772756  8731 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:33.752393 13624 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:33.752492 13599 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:33.781982  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:33.781116  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:33.665670  8477 catalog_manager.cc:5697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:33.787004  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:33.788941  8727 raft_consensus.cc:2410] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 13:03:33.793025  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.794304  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:33.769140 13639 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:33.794358  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:33.795639  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:33.805264  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:33.801259  8697 leader_election.cc:336] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.724s (SENT)
I20260709 13:03:33.807075 13125 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:03:33.808949  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46581us
I20260709 13:03:33.808768  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:33.813772  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 158208us
I20260709 13:03:33.785213 13585 raft_consensus.cc:2764] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:03:33.778791 13705 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:33.809806 13340 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:33.827039 13688 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.184s	user 0.000s	sys 0.002s
I20260709 13:03:33.833611 13568 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:33.834214  9296 leader_election.cc:336] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.776s (SENT)
I20260709 13:03:33.843104  8726 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:33.777087 13676 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:33.851632  8727 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:33.787653 13428 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:03:33.861749  9297 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:33.866926  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:33.868474  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:33.854907 13400 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:33.869057 13683 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.744s	user 0.012s	sys 0.005s
I20260709 13:03:33.847546 13705 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:33.865463 13546 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.708s	user 0.012s	sys 0.007s
I20260709 13:03:33.854633 13340 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:33.859469 13617 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.789s	user 0.013s	sys 0.007s
W20260709 13:03:33.869304  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 68098.6us
I20260709 13:03:33.895780  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:33.886143  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51965.1us
W20260709 13:03:33.900226  8697 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.748s (SENT)
I20260709 13:03:33.900658  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:33.868822 13690 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:33.868583 13247 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:33.878309 13242 raft_consensus.cc:2764] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:03:33.872041 13400 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.884989 13624 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:33.904663 13691 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.276s	user 0.000s	sys 0.002s
I20260709 13:03:33.894301 13405 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:33.902416 13546 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:33.910593  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:33.913334 13700 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 13:03:33.913477 13702 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:33.907593 13718 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:33.914678 13698 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
W20260709 13:03:33.915297 13696 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 13:03:33.868443 13599 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.916046 13568 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:33.916294 13707 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 13:03:33.916884 13701 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 13:03:33.917110 13429 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:33.869849 13713 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:33.926945 13405 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.908038  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:33.926016  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 185763us
I20260709 13:03:33.930966 13694 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:33.939893  9323 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:33.906041 13676 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:33.944382  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44235.3us
I20260709 13:03:33.908511 13247 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:33.918138 13400 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:33.898203 13683 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.948055  9325 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
W20260709 13:03:33.961257  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 180286us
I20260709 13:03:33.899331 13705 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.904867 13617 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:33.953130  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:33.955466  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 196829us
I20260709 13:03:33.955495  9323 raft_consensus.cc:606] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:33.967648  9323 raft_consensus.cc:614] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:03:33.968915  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60968.3us
I20260709 13:03:33.970499 13585 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:33.970223 13247 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.971060 13585 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:33.971519 13247 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.971951 13585 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.973516 13599 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:33.956213 13405 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.978053 13328 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:33.978494 13328 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:33.979512  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.910089  8477 catalog_manager.cc:5697] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:33.951664 13702 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:33.981308  8548 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:33.951921 13718 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:33.981505  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.983986  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:33.989383 13708 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:33.999737 13708 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:33.959049 13714 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 13:03:34.000305 13708 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:33.992527 13125 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:34.002271  8477 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 6, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 6 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:34.008400 13340 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:34.008857 13633 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.776s	user 0.007s	sys 0.004s
I20260709 13:03:34.011327 13683 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:34.011444 13705 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.014935 13617 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:33.961932 13676 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.017604 13676 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:34.018121 13622 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:34.018462 13640 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:33.979363 13328 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.026604 13328 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:33.973379 13586 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.722s	user 0.006s	sys 0.003s
I20260709 13:03:34.038221  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:34.041791  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 56146.2us
W20260709 13:03:34.047024  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 252781us
W20260709 13:03:33.993207 13516 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.840s	user 0.008s	sys 0.004s
I20260709 13:03:34.049206  9324 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:34.050177  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64357.8us
I20260709 13:03:34.053454  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.050575  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:34.054697 13638 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:34.055577 13638 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:34.056748 13638 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.013262 13695 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:03:34.072593  8549 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.934s (SENT)
I20260709 13:03:34.074290  9327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:34.074929  9327 raft_consensus.cc:606] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:34.075313  9327 raft_consensus.cc:614] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:03:34.077509  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.079510  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:34.037372 13676 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.093479 13717 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:34.099952  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:34.100366 13717 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:34.100970 13717 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:34.101305  8575 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:03:34.037106 13586 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.112696 13586 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:34.113813  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:34.004047  8477 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 6, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 6 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:34.114599  8575 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:34.115942  8477 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:34.094537 13718 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:33.998003 13702 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:34.135815  8549 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.026s (SENT)
I20260709 13:03:34.097118 13676 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.138967 13676 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:34.072917 13695 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:34.116678 13400 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.142455 13695 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.143904 13695 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:34.144152 13400 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:34.144932  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 54461us
I20260709 13:03:34.145848  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:34.009317 13633 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.148244  8575 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:34.148854  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:34.152464 13559 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:34.152619  9325 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea2142382a5d4439915e07c70837c688"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:34.175190  9325 raft_consensus.cc:606] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.175585  9325 raft_consensus.cc:614] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:34.154790 13708 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.156173 13405 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:34.159140 13709 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.338s	user 0.002s	sys 0.000s
I20260709 13:03:34.158362  8477 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 6 to 7, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 7 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:03:34.162699 13716 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.208s	user 0.000s	sys 0.004s
I20260709 13:03:34.164722 13703 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:03:34.168148  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:34.169338  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 183238us
I20260709 13:03:34.173671  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:34.178150  9327 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:34.177731  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:34.195789  9322 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:34.209457  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:34.218431  9296 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.192s (SENT)
I20260709 13:03:34.174904 13559 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:34.176636 13705 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:34.223836  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 270790us
I20260709 13:03:34.210860 13703 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:34.224424  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 160582us
W20260709 13:03:34.226855  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.217691 13247 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:34.224578  8581 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:34.235145  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:34.238488  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:34.241679  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 73934.9us
I20260709 13:03:34.242759  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:34.249212  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.250337  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:34.250561  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 171148us
I20260709 13:03:34.250954  8580 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:34.251366  8697 leader_election.cc:400] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:34.252202  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:34.252774  8583 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:34.255016  9322 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:03:34.256435  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.258397  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:34.231920 13642 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.247531 13676 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:34.267122  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.267763  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.277670  8656 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:34.268440  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.269460  8652 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:34.270076  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.267762  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.279464  8651 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.252717 13694 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:34.248874 13247 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:34.280439  8650 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:34.263924 13559 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.290380 13559 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:34.294013 13247 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.234575 13400 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 13:03:34.298563  8655 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:03:34.298905  8651 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:34.254252 13328 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.247824 13711 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 13:03:34.260017 13723 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.263942 13712 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.274312 13516 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.286337 13725 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:34.272037 13714 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.293488 13638 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.276469 13642 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:34.332118  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65107.3us
I20260709 13:03:34.335536  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:34.331125  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 88446.7us
I20260709 13:03:34.271413 13703 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.338928  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:34.334146  8650 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:34.342489  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:34.325322 13713 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.347951  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.314133 13328 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.349547  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.325536 13516 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.334255 13633 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:34.352921  8656 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:34.332733 13639 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:34.348620  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.349751  8655 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:34.349457  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.351380  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.350267  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:34.351673  8653 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:34.355554  8652 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.348704  8651 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:34.355891  8659 raft_consensus.cc:2410] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:03:34.343901 13694 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.357946 13694 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.333810 13714 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.352356 13713 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.357861  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:34.365464  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.321303 13723 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:34.399403  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54234.6us
I20260709 13:03:34.347934 13676 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:34.410343  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60153us
I20260709 13:03:34.411567  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.408947 13516 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:34.413743  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.404819  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.416518  8654 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:34.404913 13741 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:34.420524 13516 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:34.456009  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:34.401350 13736 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:03:34.426079 13717 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.427210 13721 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:34.339227 13638 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.432652  8655 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:34.442687  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:34.479493  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.351944 13738 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:34.354961 13639 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.454005  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.457082  9327 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:34.458238  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:34.477618  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:03:34.489645  9330 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.460255 13739 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:34.460382  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 55641.6us
I20260709 13:03:34.323891 13712 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.405135 13703 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:34.403982 13725 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.470890  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:34.471668  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.491783  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:34.476399  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:34.417541 13711 raft_consensus.cc:2764] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:03:34.477023  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:34.477036  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 65545.6us
I20260709 13:03:34.487802  8651 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:34.479625  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:34.499742  8659 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.475826 13721 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:34.455305 13708 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:34.511742  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.519548  8652 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.463558 13400 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:34.509828  8550 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.522570  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.522912  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:34.523180  8733 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:34.476154 13742 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:34.462847 13633 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 13:03:34.524037  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:34.495114 13639 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:34.499293 13516 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:34.488946 13738 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.530190  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.498786 13741 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.531661  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.534191  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:34.532250  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:34.542222  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.542855  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.584271  8732 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:34.576371  8728 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 8: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:34.587826  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.587867  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.588982  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.590126  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.589162  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.593649  8728 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:34.593976  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.579778  8733 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:34.577822  8724 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:03:34.546303  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.541801  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:34.594504  8732 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:34.517997 13676 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:34.599723 13740 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:03:34.511427 13725 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.574756  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
W20260709 13:03:34.639997  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65331.6us
W20260709 13:03:34.637532  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 127787us
I20260709 13:03:34.641431  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.643775  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:34.644526 13750 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:34.645591 13617 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:34.646023 13617 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.646317  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:34.647082  9325 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:34.635622 13676 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.543087 13516 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.619926  8726 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:34.575206 13736 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:03:34.543793 13742 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:34.626049 13713 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:34.628389 13712 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:34.628713  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.622769  8657 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:34.612699  8659 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:34.656414  8651 raft_consensus.cc:2376] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11. Re-sending same reply.
I20260709 13:03:34.586242 13708 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:34.574388 13738 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.637140  8728 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:34.646502 13568 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:03:34.615268  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 158423us
I20260709 13:03:34.648865 13676 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:34.648103 13750 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:34.681550  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2700) took 1480 ms. Trace:
I20260709 13:03:34.681696 13750 consensus_queue.cc:1243] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:34.682279 13750 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:34.648904 13683 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.008s
I20260709 13:03:34.682694 13750 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:34.683272 13706 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:34.683660 13706 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:34.683720 13750 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.684564 13706 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.685277 13750 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:34.651208  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:34.686088  9326 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:34.687172  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:34.687953  9326 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:34.688710  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:34.646363 13617 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:34.699388  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.700027  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2701) took 1498 ms. Trace:
I20260709 13:03:34.700443  8730 rpcz_store.cc:276] 0709 13:03:33.201286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.215606 (+ 14320us) service_pool.cc:224] Handling call
0709 13:03:33.250296 (+ 34690us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:33.265309 (+ 15013us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:33.265371 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:33.304085 (+ 38714us) log.cc:844] Serialized 26 byte log entry
0709 13:03:33.729637 (+425552us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:34.158458 (+428821us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:34.221961 (+ 63503us) log.cc:844] Serialized 14 byte log entry
0709 13:03:34.675321 (+453360us) spinlock_profiling.cc:243] Waited 9.92 ms on lock 0x7b1800313720. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287
0709 13:03:34.675446 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:34.675529 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:34.692226 (+ 16697us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b240038fac8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b67fb0 00007f14097de488 00007f14097debe8 00007f14097b6398 00007f14097b176a 00007f14097b0bb6
0709 13:03:34.699816 (+  7590us) raft_consensus.cc:1685] finished
0709 13:03:34.699837 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:34.699970 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":344319,"prepare.queue_time_us":367343,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":58266,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":22164,"spinlock_wait_cycles":46047616,"thread_start_us":342965,"threads_started":3,"wal-append.queue_time_us":446217}
W20260709 13:03:34.703764  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:34.704210  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:34.706146 13748 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:03:34.709017  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.709733  8732 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:34.715469 13747 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 13:03:34.720461 13706 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:34.722199 13742 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.681989  8729 rpcz_store.cc:276] 0709 13:03:33.200962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.208575 (+  7613us) service_pool.cc:224] Handling call
0709 13:03:33.208814 (+   239us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:33.217442 (+  8628us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:33.231140 (+ 13698us) log.cc:844] Serialized 14 byte log entry
0709 13:03:33.252805 (+ 21665us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b180002f3e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:33.615576 (+362771us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:33.616270 (+   694us) log.cc:844] Serialized 26 byte log entry
0709 13:03:33.616400 (+   130us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:34.029107 (+412707us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b180002f3e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:03:34.227468 (+198361us) log.cc:844] Serialized 14 byte log entry
0709 13:03:34.251775 (+ 24307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:34.681216 (+429441us) spinlock_profiling.cc:243] Waited 8.22 ms on lock 0x7b54000eae54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:34.681264 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:34.681301 (+    37us) raft_consensus.cc:1685] finished
0709 13:03:34.681313 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:34.681469 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124240,"prepare.queue_time_us":464593,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":102,"raft.queue_time_us":76624,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":28156,"spinlock_wait_cycles":37564416,"thread_start_us":360365,"threads_started":2,"wal-append.queue_time_us":479971}
I20260709 13:03:34.725535 13328 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:34.726531  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:34.727285  9323 raft_consensus.cc:2368] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:34.633962 13743 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:34.729779 13743 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.732000 13743 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:34.733700 13725 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:34.734418  8623 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:34.734673 13714 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:34.735242 13714 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:34.585939 13741 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.737643 13741 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:34.740571 13405 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:34.744957  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.746207  8728 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:34.650651  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.674819 13717 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:34.675765  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.750486  8651 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.676438  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.751108  8656 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:34.677062  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.679582  8726 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:34.660470  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.682134  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.752610  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.758169  8729 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.748121  9326 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 8: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:34.669131  8731 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:34.658705  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:34.655093 13516 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:34.774842  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.778267  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.752071  8653 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:34.779278  8659 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 8: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.780046  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.780822  8656 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.757197  8655 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.757457  8725 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.753611  8697 leader_election.cc:400] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:34.783602  8697 leader_election.cc:403] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:03:34.790181  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.793721  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.795056  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:34.670756 13642 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:34.747776 13741 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.798278  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.803395  8653 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.803974  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.798846  8724 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 13:03:34.801025  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.804072  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:34.670461 13497 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.119s
I20260709 13:03:34.809692  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.807374  8730 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:34.811085  8653 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:03:34.813239  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.799692  9325 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:34.801390  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.814801  8656 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:34.824018  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.826246  9297 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:34.827745  8651 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:34.828751  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.832118 13723 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:34.837615 13640 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:34.844483  8658 raft_consensus.cc:1217] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Deduplicated request from leader. Original: 10.9->[12.10-12.10]   Dedup: 12.10->[]
I20260709 13:03:34.845141  8658 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260709 13:03:34.845724  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 195114us
I20260709 13:03:34.847237  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.820627  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.848120  8659 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:03:34.849740  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.851256  8656 raft_consensus.cc:2410] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:03:34.851372  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.797667 13328 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.859459  9330 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:34.750238 13694 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.798000 13752 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:34.819489  8732 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:34.861532  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.861877  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:34.868800  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:34.863849  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 205203us
I20260709 13:03:34.869530  8659 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:34.806941 13717 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:34.860445  8654 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:34.870384  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:34.874495  9325 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:34.874564  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:34.871189  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:34.871663  8576 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:34.876174  8576 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:34.876538  8583 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:34.877585  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:34.872020  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:34.651667 13721 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.878566  9328 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:34.883481  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:34.872313 13754 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.111s	user 0.000s	sys 0.005s
I20260709 13:03:34.863660 13638 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:34.867877 13712 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:34.811158 13741 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.889309  8576 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:34.873576 13242 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:34.879177 13723 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:34.903216  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.882578 13739 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:34.904893  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:34.917390  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:34.988456  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:34.988935  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71623.2us
I20260709 13:03:34.866776 13328 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:34.918301  8576 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:34.918954  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.990564  8659 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:34.919878  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.991068  8656 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:34.920825  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.896569 13708 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:34.921926  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:34.996363  8575 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:34.923570  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:34.997351  8581 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:34.924760  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:34.982460  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:34.987190 13622 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:35.004601  8581 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:35.005015  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.009980  8582 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:35.011055  8658 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:35.011648  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 106833us
I20260709 13:03:35.011981  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:35.016505  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.020249  8583 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:35.010329  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:35.022604  8659 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:35.025269  8656 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:35.027752 13642 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:35.028069  8581 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:35.028412 13642 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.027067  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.030611  8581 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:35.031344 13400 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:35.030638  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.034832 13328 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.035805 13642 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.019320 13622 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.042690 13516 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.032678  8656 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:35.053573  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:35.054831  8729 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:35.054982  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:35.055706  8577 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:35.056653  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
W20260709 13:03:35.062348  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68745.8us
I20260709 13:03:35.063198  8576 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 13:03:35.063606  8576 pending_rounds.cc:85] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 4
I20260709 13:03:35.063838  8576 pending_rounds.cc:107] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 13:03:35.064095  8576 raft_consensus.cc:2889] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:34.995921 13723 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:34.997839 13683 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:35.066229  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:34.985235 13739 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:35.070389  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:34.995558 13638 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.071540  8580 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:34.929224 13676 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.086270  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.093602  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:35.097355  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70223.4us
I20260709 13:03:35.098378  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:35.090008  8729 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:03:35.090093  8659 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:35.105260  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.105568  8654 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:35.105983  8658 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:35.106787  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:35.069797 13703 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.555s	user 0.007s	sys 0.002s
I20260709 13:03:34.985077 13617 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.116719 12171 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:35.113116  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:35.115061  8727 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:35.121608  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:35.122428  8724 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:35.091542 13739 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:35.123678 13713 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 13:03:35.136101 13713 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:35.136521 13713 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:35.136854 13713 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:35.118770 13328 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 13:03:35.138095 13741 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:35.138427 13328 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:35.138805 13328 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:35.139600 13328 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.140758 13328 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:35.089610 13708 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:35.025046 13639 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.144232  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:35.096352 13683 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:35.114583 13752 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.119196 13642 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:35.153429  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:35.146270  8725 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:35.164094  8733 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:03:35.124590 13568 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:35.159138  8655 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:35.142299 13708 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.101799 13638 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:35.168579  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 45897.3us
W20260709 13:03:35.166742  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 148460us
I20260709 13:03:35.169509  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:35.178539  8625 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:35.180542  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:35.158339 13705 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:35.164465 13743 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:35.185074  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:35.192889  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 89289.9us
I20260709 13:03:35.165207 13642 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:35.194816  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:35.205235  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 188839us
I20260709 13:03:35.183691 13585 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:35.176349 13568 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:35.190430 13516 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.231814  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:35.182756 13703 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:35.241755  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:35.244185  8725 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:35.197247 13718 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:35.200863 13642 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:35.165103 13752 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:35.260996  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58993.7us
I20260709 13:03:35.262558  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:35.263257  8725 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:03:35.195740  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 139190us
I20260709 13:03:35.181705 13738 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.265560  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:35.265832  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:35.266665  8625 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:35.272665  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:35.275483  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 100126us
I20260709 13:03:35.235065 13741 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:35.275503  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 153943us
I20260709 13:03:35.290138  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.259621 13400 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:35.280452 13750 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.530s	user 0.010s	sys 0.000s
I20260709 13:03:35.183863 13723 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.296355  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.299160  8578 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:35.330411  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:35.348569  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 82007.7us
I20260709 13:03:35.352217  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:35.238236 13766 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
W20260709 13:03:35.363627  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 154710us
W20260709 13:03:35.364396  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:35.281618 13742 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:03:35.353852  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57559.9us
I20260709 13:03:35.369848  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:35.370162  9323 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:35.282473 13761 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:35.364972  9296 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.909s (SENT)
I20260709 13:03:35.297683 13516 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:35.385632  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.349314 13718 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:35.275851 13683 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.295172 13764 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:35.410175 13585 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:35.370570 13714 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:35.371613 13742 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:35.288995 13741 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.297525 13712 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260709 13:03:35.387598 13568 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.370901 13676 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:35.402851 13750 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:35.446548  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 180773us
W20260709 13:03:35.445327  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 81658.5us
I20260709 13:03:35.392303 13767 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:35.463011  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 98060us
I20260709 13:03:35.465967  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:35.467983  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:35.468535  9322 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:35.456480  8727 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:35.470845  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:35.471514  8727 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:35.467314  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:35.427318 13639 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.496687  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:35.426986 13768 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:35.442432 13742 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:35.428941 13764 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:35.475348  9326 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:35.507480  8727 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:35.496697  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:35.446028 13714 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:35.520648  9322 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:35.522581 13762 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:35.522961  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:35.523175 13762 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:35.523732 13762 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:35.527045 13714 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.453270 13676 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:35.535954  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.477816 13750 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:35.472746 13703 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.554502  8652 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:35.554630  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.425289 13400 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.539145 13751 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:35.567739  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.553246 13761 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:35.556947 13638 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:35.579476  9296 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.549613 13713 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.555752 13766 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 13:03:35.561911 13703 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.573225 13768 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:35.559079 13772 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:03:35.547410 13676 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.596936 13705 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:03:35.623494 13771 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:03:35.625104 13739 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.631735 13703 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.570734 13712 consensus_queue.cc:1243] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:35.579881 13751 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:35.647311 13705 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:35.597767 13761 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:35.590983 13752 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:35.649446 13738 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:35.696558 13741 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.713888  9324 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:35.702752 13761 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:35.673357 13767 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:35.651314 13750 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:35.729867 13773 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 13:03:35.651187 13764 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:35.739511  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:35.731983 13516 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:03:35.744827  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 165438us
I20260709 13:03:35.746279  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:35.716701 13705 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.723115 13751 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.751765  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:35.753691  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.755764 13787 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:35.760825 13739 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.763110 13617 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.764717  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:35.765457 13713 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:35.766167  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:35.767099  8726 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:35.770179 13622 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:35.751255 13772 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:35.760924 13787 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:35.774065  8732 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:35.783339 13768 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:35.786839  9296 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.794384 13622 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:35.798983  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.805505  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 888) took 1147 ms. Trace:
I20260709 13:03:35.788276 13787 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:35.810473  8730 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:35.813675 13622 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:35.814620 13676 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:35.815254 13676 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:35.815605 13642 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.000s
I20260709 13:03:35.815788 13676 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.816082 13642 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.817242  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2460) took 1287 ms. Trace:
I20260709 13:03:35.817270  8725 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:35.817754  8733 rpcz_store.cc:276] 0709 13:03:34.529864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:34.596647 (+ 66783us) service_pool.cc:224] Handling call
0709 13:03:34.596980 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:35.164556 (+567576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:35.816953 (+652397us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b5400229a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:35.817179 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1294267264}
W20260709 13:03:35.813205  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:35.805986  9323 rpcz_store.cc:276] 0709 13:03:34.658577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:34.727811 (+ 69234us) service_pool.cc:224] Handling call
0709 13:03:34.728071 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:35.370665 (+642594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:35.805251 (+434586us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54002116d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:35.805436 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1475631872}
I20260709 13:03:35.832528  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:35.833250  8655 raft_consensus.cc:2376] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11. Re-sending same reply.
I20260709 13:03:35.837250  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:35.839764  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:35.835623  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:35.842536  8697 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:35.800148 13772 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:35.800817 13768 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:35.846601 13340 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:35.852030 13340 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:35.852500 13340 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:35.853976 13783 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 13:03:35.847380  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:35.856197  9330 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:35.858034  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:35.858752  9323 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:35.863613 13742 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:35.863425 13340 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.864360 13694 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.849833 13713 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:35.871979  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:35.872627  9323 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:35.874836 13568 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:35.877872  9322 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:35.878739  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:35.879230  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:35.880576  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.903357  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.893134  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 816) took 1359 ms. Trace:
I20260709 13:03:35.904947  8652 rpcz_store.cc:276] 0709 13:03:34.534018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:34.541993 (+  7975us) service_pool.cc:224] Handling call
0709 13:03:35.892901 (+1350908us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:35.893061 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1257609088}
I20260709 13:03:35.907976  8652 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:35.908191  8728 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:35.909152  8659 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
W20260709 13:03:35.910104  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:35.910742  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.913472  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:35.914674  8655 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:35.915485  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:35.911397  8659 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:35.922703  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.923925  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.924899  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.928467  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:35.857363 13772 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.876989 13713 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:35.936307  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.891078 13622 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:35.935861  8657 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:35.906683 13617 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:35.884268 13750 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:35.883476 13739 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:35.881477 13585 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:35.934242  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.871552 13751 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 13:03:35.921725 13781 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:35.958483 13713 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.932158 13712 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:35.959653 13791 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 13:03:35.959928 13712 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:35.960438 13712 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:35.962559 13762 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.964035  9323 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:35.931984 13723 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:35.965593 13723 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:35.965854  9330 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:35.966233 13723 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:35.967557  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:35.968281  8728 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:35.953820 13694 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260709 13:03:35.959081 13751 consensus_queue.cc:1243] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:35.972775 13568 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:35.973052  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:35.973994 13617 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:35.976096 13617 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:35.976735 13617 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:35.977751 13328 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.978286 13328 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.979353 13762 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:35.979955 13762 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:35.980408 13762 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.981981 13617 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.983242 13781 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:35.983459  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:35.983791 13781 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:35.981165 13328 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:35.929695 13704 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 13:03:35.987962 13751 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:35.954136 13739 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:35.988531 13751 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:35.988739 13739 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:35.924191 13767 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 13:03:35.990804  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.991889  8728 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
W20260709 13:03:35.992355  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 81220.1us
I20260709 13:03:35.993716  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:35.941320 13780 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:35.936486 13428 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 13:03:35.953042 13742 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:36.004429  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:35.989589 13751 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.006110 13676 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.007299  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:36.008664 13793 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:03:36.015645  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.032050  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.035804  8728 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:36.036087 13766 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:36.036834 13766 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:36.012634 13568 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:36.038904 13405 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:36.039359 13405 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:36.040207 13405 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.041788  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:36.042414  8727 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.042711 13640 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:36.043234 13640 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.017488 13751 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:36.049970  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:36.050527  9331 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.051476 13340 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.049005 13516 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 13:03:36.054659  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:36.034009  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.059821  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:36.021003 13712 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.000810 13780 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:36.028308  9323 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:36.029027 13776 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:36.003801 13768 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:36.030305 13742 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:36.061014  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.061508  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.016585  9328 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:36.058854  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:36.063656 13749 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.065438  8652 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:36.072464  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:36.065724  8724 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:36.065990 13633 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:36.076589  8730 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:36.080631  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.063728 12171 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 13:03:36.079612  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.087205  8659 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:36.067360 13787 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.065214  8729 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:36.065155  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:35.999188 13750 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:36.072320  9325 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:36.073205  8727 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:36.084623 13752 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:36.099181 13752 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:36.099550  8727 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:36.099915  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.074422 13711 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:03:36.084889 13717 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.101177  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:36.101563 13711 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:36.102005 13711 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:36.102847 13751 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:03:36.077162 13792 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 13:03:36.103039 13711 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.079595  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.079103  8652 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:36.101958  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.105301  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:36.108546  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.116614  8656 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.114428  8733 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 13:03:36.117580  8733 pending_rounds.cc:85] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:03:36.106735  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:36.105218  8580 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:36.117851  8724 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:36.117939  8733 pending_rounds.cc:107] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 13:03:36.119637  8733 raft_consensus.cc:2889] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:36.116871  8651 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:36.121273  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.121965  8651 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:36.045298 13640 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:36.062716 13405 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.123466  8732 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:36.126463  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.108960  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.127357  8650 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:36.124727  9328 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:36.116525  8652 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.128151  8652 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:36.106769 13723 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.108228 13627 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:03:36.129264  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.110004 13242 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 13:03:36.129895  8659 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:36.133952 13787 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:36.105481 13781 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:36.105723 13762 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:36.135726  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.099565 13752 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.137622 13725 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:36.138492 13725 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:36.140547 13784 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:36.141626 13765 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:36.101230 13717 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.104403 13711 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:36.144069 13627 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:36.144562 13627 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:36.145653 13627 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.147233 13627 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:36.141206 13784 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:36.151000 13784 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.152172  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.152268  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.153012  8653 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.153640 13743 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:36.154356 13723 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:36.155857  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.157291 13767 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:36.157763 13767 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.158147 13767 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.158177  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.162918 13400 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:36.163789  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.164093 13741 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.164655 13741 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.163931 13242 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:36.137147  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.171862  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:36.173273  8652 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 13:03:36.173895  8652 pending_rounds.cc:85] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 5
I20260709 13:03:36.171105  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.174499  8656 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:36.165416 13328 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:36.172850  9326 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:36.165380  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.177300  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:36.178376  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.174526  8652 pending_rounds.cc:107] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 13:03:36.176110 13328 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.181092 13328 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.181387 13743 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:36.181874 13743 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.182300 13743 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.185222 13721 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.166617 13741 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.167099 13242 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:36.188174 13242 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:36.190622 13705 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:36.191697 13717 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:36.072845 13780 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.189239 13242 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.195847 13242 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:36.084728 13683 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:36.197746 13340 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:36.197871 13683 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.199230 13642 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:36.073045 13776 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:36.202100  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:36.202193 13776 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.114931 13768 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:36.204442 13768 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.204972  9326 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:36.180496  8652 raft_consensus.cc:2889] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:36.202433  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.198475 13340 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.210014  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:36.228380  8659 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:36.230219  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:36.236657  8732 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:36.241899  8733 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:36.237378  8653 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:36.246320  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.250406  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.250762  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.251989  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:36.256516  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46592.6us
W20260709 13:03:36.258317 13692 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 13:03:36.258909 13717 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:36.259400 13717 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.259222 13779 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:36.260777  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.261582 13725 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:36.258900 13639 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 13:03:36.259877 13717 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:36.263067 13767 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.234051 13642 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:36.265445 13642 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.265898 13328 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.267645  9328 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:36.267768 13642 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.268635 13328 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.263283 13725 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:36.272507  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.271629  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.278294  8579 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:36.279011  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.255498 13723 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:36.272779  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.234213 13765 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:36.270644  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.270624  8725 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:36.263407 13777 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:36.272222  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.324796  8577 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:36.271543  8732 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:36.272449  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.326363  8583 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.275481  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:36.277006  8730 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:36.327584  9328 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.272915  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.276330  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.270720  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.280380  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.331275  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.280638  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.333266  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.206636 13742 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:36.335189 13328 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:36.336042 13328 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:36.337574  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.249770  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.298273 13767 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:36.219424 13683 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:36.254364 13749 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:36.302217 13727 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.301641 13750 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.233283 13340 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:36.307983  9331 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:36.311172  8731 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:36.386595  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.387107  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:36.388338  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 51177.2us
I20260709 13:03:36.312134  8659 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:03:36.323669  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:36.338398  8576 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:36.360919  8580 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:36.360360 12171 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:36.391597  9330 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:36.391952 12171 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.356631  8724 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:03:36.336805  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 60614.6us
I20260709 13:03:36.392637  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.338536 13777 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.397571  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.398257  8656 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:36.359951  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 432111us
I20260709 13:03:36.380167  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:36.383735  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:36.322980 13797 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 13:03:36.413066 12171 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:36.405241  8579 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:36.398729  9329 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:36.362984  8653 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:36.261171 13779 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:03:36.337358 13627 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.396696  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.392521  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:36.425925  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.416066  8578 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:36.359280 13723 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:36.376111 13692 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:36.427137  8650 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.434731  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.435217  8582 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:36.435680  8652 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:36.437121  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.444118 13639 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:36.445971 13702 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.446566 13702 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.447901  8575 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:03:36.448895 13702 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:36.449384 13723 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.445185  8582 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:36.398473 13718 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:36.397922 13776 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:36.458191  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.414688 13777 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.463409  9328 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:36.468182  8656 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:03:36.436087 13793 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:03:36.380041 13721 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 13:03:36.415673 13767 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.382402 13752 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.474772  8650 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:36.475831  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:36.475478  8583 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:36.476171  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.455181 13787 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.477092  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.478616 13718 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:36.453424 13627 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.387282 13727 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.492825  8652 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:36.460871 13568 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:03:36.523138  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47031.7us
W20260709 13:03:36.523296  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47547.1us
I20260709 13:03:36.493638 13713 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:36.535835  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:36.456753 13765 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 13:03:36.455776 13751 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 13:03:36.526371  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:36.470264 13742 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 13:03:36.474766 13704 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
W20260709 13:03:36.535756  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 478693us
I20260709 13:03:36.487718 13739 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.489499 13717 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.532197 13741 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:36.540648 13516 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:36.607331  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 168692us
I20260709 13:03:36.485816 13328 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.528641 13787 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.538403 13725 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:36.528076 13428 raft_consensus.cc:2764] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:03:36.545298 13762 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:36.658090  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 117608us
I20260709 13:03:36.550045 13642 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:36.501048 13767 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.661099  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:36.538913 13568 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.456115 13692 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.559443 13585 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:36.674130  8584 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:36.674672  8727 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:36.530047 13743 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:36.677573  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:36.675680 12171 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:36.563789 13714 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.116s
I20260709 13:03:36.668959 13617 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:36.745430 12171 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:36.671412 13681 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:36.677280 13516 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:36.748181 13725 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.673817 13766 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 13:03:36.726549 13751 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:36.729067 13568 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.670981 13713 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.104s
W20260709 13:03:36.900949  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 239292us
I20260709 13:03:36.774449 13681 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:36.916263  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.762717 13702 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20260709 13:03:36.785547 13617 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.939205  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:36.940011  8725 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:36.940150  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.941257  8726 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:36.941983 13799 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 13:03:36.942500  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:36.943579 13627 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.944167 13516 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.945660 13706 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:36.931888 13828 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.946673 13828 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.948833 13828 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.952471 13752 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:36.956995  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.760363 13718 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:36.960480  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:36.936548  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:36.961211  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:36.960932  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:36.953502 13801 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 13:03:36.935047 13723 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:36.933583 13780 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.738s	user 0.009s	sys 0.004s
I20260709 13:03:36.962313  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.974308  8653 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:36.902870 13751 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:36.968000  8654 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:36.968202  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:36.965934  8725 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:36.980967  8576 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:36.968547 13640 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:36.962330 13807 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 13:03:36.987417  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:36.928884 13764 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
W20260709 13:03:37.013338  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50052.8us
I20260709 13:03:37.013703  8652 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:03:37.018868  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:37.033872  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:37.038549  8732 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:37.027135  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:37.027660 13809 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.298s	user 0.001s	sys 0.000s
W20260709 13:03:37.074662  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 52182.2us
I20260709 13:03:37.026587 13742 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.994715 13828 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 13:03:36.985915 13780 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.055591 13717 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.062204 13640 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:37.061676 13702 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.986969 13681 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.091853 13825 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:37.086400 13787 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.115689  9330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "266236cfcd824409b757505321b36d81"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:37.129993  9330 raft_consensus.cc:606] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:37.130723  9330 raft_consensus.cc:614] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:03:37.136022 13804 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 13:03:37.137208 13824 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:37.138414 13824 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:37.079371 13703 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:03:37.069906 13706 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:03:37.140071 13400 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:37.139539 13824 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.144447 13806 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.158s
I20260709 13:03:37.145176 13830 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:37.201653 13400 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:37.202407 13639 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.202967 13639 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.203635 13400 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:37.152850 13814 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.243s	user 0.000s	sys 0.000s
I20260709 13:03:37.205433 13400 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:37.157824 13751 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.118315 13743 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.158524  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 899) took 1206 ms. Trace:
W20260709 13:03:37.207041 13817 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.464s	user 0.002s	sys 0.000s
I20260709 13:03:37.206914  9322 rpcz_store.cc:276] 0709 13:03:35.951697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.951907 (+   210us) service_pool.cc:224] Handling call
0709 13:03:35.952394 (+   487us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:35.954197 (+  1803us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:35.954227 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:35.958357 (+  4130us) log.cc:844] Serialized 26 byte log entry
0709 13:03:36.036353 (+ 77996us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b180046a6e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:36.461651 (+425298us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:36.461691 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.461781 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:37.141114 (+679333us) raft_consensus.cc:1685] finished
0709 13:03:37.141139 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:37.158441 (+ 17302us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1319,"raft.queue_time_us":2973,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":27918592,"thread_start_us":425089,"threads_started":1,"wal-append.queue_time_us":910821}
I20260709 13:03:37.107838 13767 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.156636 13780 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:37.201864 13830 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:37.210892 13739 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.210453 13830 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.211771 13830 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:37.212185  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2731) took 1105 ms. Trace:
I20260709 13:03:37.212602  8727 rpcz_store.cc:276] 0709 13:03:36.107077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.107543 (+   466us) service_pool.cc:224] Handling call
0709 13:03:36.107804 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:36.740014 (+632210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.211942 (+471928us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b540022a954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:37.212124 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1303179008}
I20260709 13:03:37.205101 13639 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:37.123155 13826 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:37.165486 13727 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.223469 13639 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.224265 13639 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.226634 13639 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.222613 13826 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:37.228134 13826 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:37.092290 13832 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:03:37.099813 13742 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.229646 13617 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:37.256744 13705 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.066s	user 0.011s	sys 0.000s
I20260709 13:03:37.257452  8658 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:37.261917 13832 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:37.262418 13832 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:37.263347 13832 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.266700 13739 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:37.267622 13803 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260709 13:03:37.267452 13739 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.270102 13784 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:37.270601 13784 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:37.271277  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2911) took 1163 ms. Trace:
I20260709 13:03:37.271483 13784 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.271705  8651 rpcz_store.cc:276] 0709 13:03:36.107910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.173915 (+ 66005us) service_pool.cc:224] Handling call
0709 13:03:36.174271 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:36.174662 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:36.174710 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:36.175308 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 13:03:36.564876 (+389568us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:36.564916 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.564994 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:37.271059 (+706065us) raft_consensus.cc:1685] finished
0709 13:03:37.271083 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:37.271220 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":258967,"threads_started":1,"wal-append.queue_time_us":1013984}
I20260709 13:03:37.272836 13784 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.276733 13742 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.282177  9322 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 13:03:37.282816 13827 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 13:03:37.284346  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 959) took 1009 ms. Trace:
I20260709 13:03:37.284816  8579 rpcz_store.cc:276] 0709 13:03:36.275015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.463990 (+188975us) service_pool.cc:224] Handling call
0709 13:03:36.464705 (+   715us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:36.474741 (+ 10036us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:36.474772 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:36.475193 (+   421us) log.cc:844] Serialized 26 byte log entry
0709 13:03:36.897414 (+422221us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:36.897459 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.897533 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:37.281896 (+384363us) spinlock_profiling.cc:243] Waited 949 us on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:37.284004 (+  2108us) raft_consensus.cc:1685] finished
0709 13:03:37.284035 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:37.284247 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9297,"raft.queue_time_us":18514,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":8037,"spinlock_wait_cycles":2181888,"thread_start_us":302598,"threads_started":1,"wal-append.queue_time_us":454051}
I20260709 13:03:37.288213  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2867) took 1014 ms. Trace:
I20260709 13:03:37.288820  8584 rpcz_store.cc:276] 0709 13:03:36.273807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.332900 (+ 59093us) service_pool.cc:224] Handling call
0709 13:03:37.287935 (+955035us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54003a3c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:37.288130 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":781636480}
I20260709 13:03:37.292001  8584 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:03:37.293375  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:37.293820  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:37.294302  8578 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.295246  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:37.296474 13712 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:37.296733 13756 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.297720 13756 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:37.242379  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:37.245956 13776 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.771s	user 0.009s	sys 0.003s
I20260709 13:03:37.299760  8728 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 13:03:37.300158  8728 pending_rounds.cc:85] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 6
I20260709 13:03:37.300391  8728 pending_rounds.cc:107] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 13:03:37.300693  8728 raft_consensus.cc:2889] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:37.246559 13829 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:37.304384 13829 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:37.305339  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:37.297884  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:37.306572  8582 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:37.307808  8584 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:03:37.122777 13808 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.246s	user 0.001s	sys 0.000s
W20260709 13:03:37.228886 13405 consensus_meta.cc:287] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.051s	user 0.008s	sys 0.004s
W20260709 13:03:37.249243 13340 consensus_meta.cc:287] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.834s	user 0.009s	sys 0.003s
I20260709 13:03:37.313205  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1168) took 1145 ms. Trace:
I20260709 13:03:37.121786 13725 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:37.318586  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 287017us
I20260709 13:03:37.127224 13825 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:37.326691  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:37.327353  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.322413  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:37.330766  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:37.165308 13242 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.969s	user 0.010s	sys 0.001s
I20260709 13:03:37.324071  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:37.313781  9324 rpcz_store.cc:276] 0709 13:03:36.168146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.168311 (+   165us) service_pool.cc:224] Handling call
0709 13:03:36.168673 (+   362us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:36.205277 (+ 36604us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:36.205600 (+   323us) log.cc:844] Serialized 14 byte log entry
0709 13:03:36.489257 (+283657us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:36.496504 (+  7247us) log.cc:844] Serialized 52 byte log entry
0709 13:03:36.559835 (+ 63331us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:36.560360 (+   525us) log.cc:844] Serialized 14 byte log entry
0709 13:03:37.247341 (+686981us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b18003ea9a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:03:37.247562 (+   221us) log.cc:844] Serialized 14 byte log entry
0709 13:03:37.286266 (+ 38704us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b18003ea9a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:37.312862 (+ 26596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.312955 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:37.312997 (+    42us) raft_consensus.cc:1685] finished
0709 13:03:37.313008 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:37.313149 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":736790,"prepare.queue_time_us":689833,"prepare.run_cpu_time_us":141,"prepare.run_wall_time_us":144,"raft.queue_time_us":39633,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":19289,"spinlock_wait_cycles":157496448,"thread_start_us":282481,"threads_started":1,"wal-append.queue_time_us":916404}
I20260709 13:03:37.335547  8579 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 13:03:37.336215  8579 pending_rounds.cc:85] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 7
I20260709 13:03:37.336512  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2473) took 1279 ms. Trace:
I20260709 13:03:37.265142 13832 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:37.125567 13692 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.316267  8578 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:37.337086  8728 rpcz_store.cc:276] 0709 13:03:36.057554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.060312 (+  2758us) service_pool.cc:224] Handling call
0709 13:03:36.060576 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:37.301175 (+1240599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.336181 (+ 35006us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400064d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:37.336432 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2848868096}
I20260709 13:03:37.348142 13825 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.349669 13833 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.349879 13749 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:37.336553  8579 pending_rounds.cc:107] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 12.8
W20260709 13:03:37.371759  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.372799  8579 raft_consensus.cc:2889] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:37.374105  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:37.357472  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 338675us
I20260709 13:03:37.374998  9322 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:37.331990 13777 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.837s	user 0.011s	sys 0.001s
I20260709 13:03:37.114838 13702 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.113267 13640 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:37.383344  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61825.3us
I20260709 13:03:37.392777  8651 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: [11.8-12.9]
I20260709 13:03:37.393033  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:37.387116  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:37.394649  8732 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:37.396152  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:37.397130  9322 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:37.400887  8732 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:37.390656  8584 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:37.403774  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:37.359592 13676 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:37.360427 13712 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:37.408658  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:37.419832  8729 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:37.409034  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:37.414364  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.422317  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.417372  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:37.417363  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:37.425053  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:37.426532  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:37.427686  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.494338  8653 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:37.495606  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:37.430471  8729 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:37.418787  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:37.305385 13829 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.424211  8658 raft_consensus.cc:2376] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 13. Re-sending same reply.
I20260709 13:03:37.437750  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:37.437749  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.438788  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:37.326231 13340 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.439136  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:37.439523  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2900) took 1384 ms. Trace:
I20260709 13:03:37.433465  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:37.504351  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 70994.6us
I20260709 13:03:37.503577  8655 rpcz_store.cc:276] 0709 13:03:36.055562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.057894 (+  2332us) service_pool.cc:224] Handling call
0709 13:03:36.058222 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:36.067423 (+  9201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:36.067467 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:36.068037 (+   570us) log.cc:844] Serialized 26 byte log entry
0709 13:03:36.488785 (+420748us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:36.488844 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.488937 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:37.439297 (+950360us) raft_consensus.cc:1685] finished
0709 13:03:37.439320 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:37.439460 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11703,"raft.queue_time_us":9979,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":2326,"thread_start_us":420147,"threads_started":1,"wal-append.queue_time_us":639750}
I20260709 13:03:37.504762 13714 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:37.505309 13714 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:37.505589  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.506448  8655 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:37.394351 13777 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.300945 13568 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.406306 13768 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.464829  8652 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:03:37.465092  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.408378 13825 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:37.474853  9329 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:37.475417  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:37.476176 13749 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.380204 13833 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.477240 13742 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.541013 13742 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.477843 13405 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.478142  9331 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:37.480180  8579 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 13:03:37.481091 13703 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:37.483155  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:37.547196  8584 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:37.547355  8577 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:37.483155  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:37.548154  9330 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:37.486001  8580 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:37.486865  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:37.549587  8576 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:37.422691 13828 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.421792 13676 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:37.425864  8732 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:37.408450 13779 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:37.502643 12171 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:37.504925  8651 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:37.505729 13714 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.379480 13497 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:03:37.442217  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.526039  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:37.526343  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 178905us
I20260709 13:03:37.522724  8729 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:37.501727 13727 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:37.529019 13841 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 13:03:37.582284 13767 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 13:03:37.598213 13739 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:37.601822 13835 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 13:03:37.602495 13764 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:37.602929 13764 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:37.491534 13756 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.507332  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:37.529299  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:37.608180  8578 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:37.423554 13683 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.078s
I20260709 13:03:37.535537  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.541378 13340 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.539297  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.612313  8652 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:37.540180  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:37.613382  8658 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.540339  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:37.616542 13833 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.611660 13692 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 13:03:37.617228 13764 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:37.620961 13723 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.643s	user 0.011s	sys 0.000s
I20260709 13:03:37.603931 13727 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.629612  8651 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:37.552067 13639 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:37.533141 13741 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.075s
I20260709 13:03:37.543274 13742 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:37.550562  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.500109  9325 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.528863 13802 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:37.553800 12171 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 13:03:37.532295 13830 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 13:03:37.609500  8650 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:37.539079 13749 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:37.689759  8624 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.008s	sys 0.037s
W20260709 13:03:37.690260  8624 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.008s	sys 0.037s
I20260709 13:03:37.433507 13694 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:37.548705  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:37.551045 13400 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260709 13:03:37.636699  8659 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:37.551467  8726 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 13:03:37.581830 13828 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.545516 13787 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.093s
I20260709 13:03:37.537581 13768 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.669210  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2291) took 1334 ms. Trace:
W20260709 13:03:37.678900  9327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.021s	sys 0.025s
W20260709 13:03:37.721988  9327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.021s	sys 0.025s
W20260709 13:03:37.681854  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.010s	sys 0.032s
I20260709 13:03:37.627595 13727 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.678545 13802 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:37.542402 13702 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.624145 13712 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:03:37.673939 13762 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.943s	user 0.009s	sys 0.003s
I20260709 13:03:37.543697 13405 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:37.669242 13843 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 13:03:37.528076 13826 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.552285 13676 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:37.628959 13779 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:37.621343 13568 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:37.722571  9322 raft_consensus.cc:606] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:37.721789  8577 rpcz_store.cc:276] 0709 13:03:36.335075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.428202 (+ 93127us) service_pool.cc:224] Handling call
0709 13:03:36.428582 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:37.551732 (+1123150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.668948 (+117216us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54003e7954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:37.669133 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2572463488}
W20260709 13:03:37.722489  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.010s	sys 0.032s
I20260709 13:03:37.668562 13777 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.740571 13400 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:37.651851  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 102116us
W20260709 13:03:37.726725  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 178045us
W20260709 13:03:37.711951 13328 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.018s	sys 0.079s
I20260709 13:03:37.674288 13640 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:37.708909  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.621526 13723 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:37.815949  8731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.051s	sys 0.058s
W20260709 13:03:37.816504  8731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.051s	sys 0.058s
I20260709 13:03:37.624538 13683 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.802345  9322 raft_consensus.cc:614] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:03:37.805619  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 783089us
I20260709 13:03:37.721274 13751 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.765810 13712 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:37.821130  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.824695  8577 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
W20260709 13:03:37.809262 13328 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.019s	sys 0.079s
I20260709 13:03:37.777503 13779 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.772589 13762 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:37.710911 13639 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.858093  8658 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:37.861436  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:37.862674  8575 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:37.817176 13568 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.816275 13400 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:37.849700 13843 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 13:03:37.846648 13683 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:37.885485 13824 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.019s	sys 0.000s
I20260709 13:03:37.852792 13779 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:37.927752 13826 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.932452 13714 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.933799 13848 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:37.933916 13405 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.934441 13848 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.935712 13829 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:37.936710 13848 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.938241 13768 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:37.939142 13639 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.939576 13806 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:37.940023 13806 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:37.901077 13742 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.942205 13765 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:37.942816 13765 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:37.940032 13828 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:37.932493  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.945595  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:37.948161 13784 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:37.948948  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.952045  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:37.952901 13497 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:37.953310 13497 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.953722 13497 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:37.953994 13400 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.957067 13725 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:37.923316 13802 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.957777 13725 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.942227 13742 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.960268 13725 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.900516 13568 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.962910 13568 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:37.923720 13822 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:37.963814 13822 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.948954 13784 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:37.965704 13784 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:37.966881 13742 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.937238 13848 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:37.847910 13751 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:37.958559 13802 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:37.983860  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:37.973932  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2916) took 1579 ms. Trace:
I20260709 13:03:37.995272  8659 rpcz_store.cc:276] 0709 13:03:36.394831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.419192 (+ 24361us) service_pool.cc:224] Handling call
0709 13:03:36.419634 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:37.715967 (+1296333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.973705 (+257738us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000596d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:37.973867 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2798252928}
W20260709 13:03:37.986779  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 455283us
I20260709 13:03:37.970799 13749 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:37.975529 13714 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.999958 13683 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:38.006186  9296 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.000s	sys 0.013s
W20260709 13:03:38.006723  9296 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.001s	sys 0.014s
W20260709 13:03:37.970151 13832 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.594s	user 0.008s	sys 0.003s
I20260709 13:03:38.012219  9325 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:38.014310  8656 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:38.014844  8656 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:38.021325  8729 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:03:38.023902  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.024901  8732 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:38.026499  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:38.028743  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:37.967677 13742 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:38.015887  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:37.967811 13784 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:38.033257  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.011s	sys 0.004s
W20260709 13:03:38.046631  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.034s	sys 0.071s
W20260709 13:03:38.046819  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.011s	sys 0.004s
W20260709 13:03:38.047082  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.034s	sys 0.071s
W20260709 13:03:38.047300  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 538862us
W20260709 13:03:38.047531  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 522150us
I20260709 13:03:38.048321  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:37.926326 13824 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.019s	sys 0.000s
I20260709 13:03:37.975164 13751 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:37.967480 13822 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:38.038869  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.008594 13830 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:38.042624  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.043109  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.050081  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:37.940943 13806 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.009052 13242 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:38.030961  8656 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.057843  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:38.058063  8725 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:38.066836  8656 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:38.058991  8732 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:38.027959 13725 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:37.968175 13848 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.064158  8730 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:38.029834 13640 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.068046  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:38.070194  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:38.116966  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 62199.3us
I20260709 13:03:38.088047  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:38.119115  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:37.943907 13765 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:38.113840  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 456222us
I20260709 13:03:38.120141  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:38.047760 13428 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:38.064065 13822 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:38.116545 13832 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:03:38.113533 13617 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.000s	sys 0.015s
I20260709 13:03:38.122689  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:38.122845  9329 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:38.123206  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:38.116285 13806 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:38.158471  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:38.157294  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 89345.6us
I20260709 13:03:38.123840  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:38.160449  9330 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
W20260709 13:03:38.124460  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66728.7us
I20260709 13:03:38.162365  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:38.124992 13405 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:38.179013  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:38.125478 13833 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.125692  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2481) took 1187 ms. Trace:
I20260709 13:03:38.126466 13848 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:38.129979 13777 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.036139 13828 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:03:38.119783  9326 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:03:38.136490  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:38.137152  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:38.085244  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:38.189375  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 52966.1us
I20260709 13:03:38.136478 13692 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.143373  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:38.119809  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.216861  8658 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 8.
I20260709 13:03:37.925251 13712 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.054137 13767 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.127s
W20260709 13:03:38.222335  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 85240.9us
I20260709 13:03:38.224141  8696 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:38.150074  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.151368  8656 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 13:03:38.225291  8656 pending_rounds.cc:85] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 8
I20260709 13:03:38.152122 13639 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.225600  8656 pending_rounds.cc:107] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 13:03:38.152346 13822 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.225942  8656 raft_consensus.cc:2889] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:38.151908 13723 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.227317  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.227986  8656 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:03:38.152395  8650 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:03:38.154232  9327 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEARNER]: Advancing to term 11
W20260709 13:03:38.167097  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:38.051702 13568 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 13:03:38.235368  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:38.152608  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:38.162415  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:38.180892  8657 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:03:38.179976  8731 rpcz_store.cc:276] 0709 13:03:36.938306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.938436 (+   130us) service_pool.cc:224] Handling call
0709 13:03:36.968477 (+ 30041us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:36.969602 (+  1125us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:36.969643 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:37.124026 (+154383us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b2c000b2cd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:37.124624 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 13:03:37.817050 (+692426us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:37.817333 (+   283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.817456 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:38.125362 (+307906us) raft_consensus.cc:1685] finished
0709 13:03:38.125386 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:38.125629 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":670,"raft.queue_time_us":31719,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":12583,"spinlock_wait_cycles":3237632,"thread_start_us":691425,"threads_started":1,"wal-append.queue_time_us":949521}
I20260709 13:03:38.186275  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:38.117894 13772 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.085363  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:38.187479  9297 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.003s	sys 0.023s
W20260709 13:03:38.153018 13617 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.000s	sys 0.016s
I20260709 13:03:38.190901  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:38.142932  8651 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 13:03:38.328140  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 137285us
W20260709 13:03:38.193311  8724 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.009s	sys 0.014s
W20260709 13:03:38.329008  8724 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.009s	sys 0.014s
W20260709 13:03:38.193475  9295 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.003s	sys 0.020s
W20260709 13:03:38.329648  9297 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.830s	user 0.003s	sys 0.024s
W20260709 13:03:38.329818  9295 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.003s	sys 0.021s
I20260709 13:03:38.210971 13676 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.206965  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:38.052162 13742 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.180074 13633 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:38.060729 13784 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.120s
I20260709 13:03:38.138214 13428 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:38.228996  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.065179 13830 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.355811  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:38.359057  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2742) took 1322 ms. Trace:
W20260709 13:03:38.359694  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:38.359683  8730 rpcz_store.cc:276] 0709 13:03:37.037168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.037377 (+   209us) service_pool.cc:224] Handling call
0709 13:03:37.037814 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:38.186851 (+1149037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:38.358704 (+171853us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002044d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:38.358976 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2351402752}
I20260709 13:03:38.239302  8725 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:38.232962 13639 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.242287 13723 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.218490 13848 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:38.213903 13750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.913s	user 0.020s	sys 0.006s
I20260709 13:03:38.290473 12171 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:38.286520  8659 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:38.215500 13832 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.216496 13749 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:38.141652 13640 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.229450 13831 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:38.355687 13756 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:38.228127 13739 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:38.233218 13822 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:38.237489  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.396742  8732 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:03:38.373523 13750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.073s	user 0.026s	sys 0.007s
I20260709 13:03:38.322880  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:38.322975  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 476651us
I20260709 13:03:38.185890 13833 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.228577 13516 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.099s
I20260709 13:03:38.341336  8731 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:38.341257  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:38.342315  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 135484us
I20260709 13:03:38.212970 13777 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.319355 13824 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:38.345732  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.351846  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:38.321517 13328 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:38.120471 13725 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:38.339443 13825 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.801s	user 0.010s	sys 0.003s
I20260709 13:03:38.298133 13779 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.230073 13692 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.396206 13756 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.374558  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.420920  8730 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:38.374841  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:38.340269 13633 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:38.380057  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 251612us
I20260709 13:03:38.387766  8659 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:38.426620  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:38.388715  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:38.241243 13718 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 13:03:38.318746 13859 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 13:03:38.375914 13848 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.401935  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:38.405191  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:38.463908  8731 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:38.338480 13772 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.408185  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:38.407194 12171 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 13:03:38.406733  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:38.406661  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:38.468015  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61452.7us
W20260709 13:03:38.406172  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 165238us
I20260709 13:03:38.357620 13497 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.391598 13831 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:38.412611  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:38.395422 13828 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:38.355020 13428 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:38.415167  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 179887us
I20260709 13:03:38.417951  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.415819  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:38.432173  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:38.503216  8656 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:38.433444  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.407202 13824 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:38.438620  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:38.415704 13749 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:03:38.431500 13242 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.448555 13405 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:38.458811 13639 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:38.445109 13739 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:38.459510 13779 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:38.428748 13864 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:38.428944 13633 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:38.460621 13676 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:38.460997 13692 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:38.462648 13725 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:38.477595  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.427070 13756 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:38.415239 13866 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:38.394122 13832 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:03:38.476826 13858 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:38.544952 13864 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:38.491106  8731 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:38.497264  9330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:38.429874 13776 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:03:38.498337  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 91677us
W20260709 13:03:38.504626  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:38.581915  9330 raft_consensus.cc:606] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:38.582389  9330 raft_consensus.cc:614] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:38.415512 13750 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 13:03:38.502902  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:38.506778  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68282.6us
I20260709 13:03:38.428107  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:38.585644 13405 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.507592 13842 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:38.587064 13842 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.587548 13842 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:38.459372 13742 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:38.591990 13742 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:38.468235 13848 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:38.509266  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.511435  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:38.599653  8656 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:03:38.602283  8656 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
W20260709 13:03:38.516887  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 364332us
I20260709 13:03:38.486855 13825 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:38.604543 13825 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:38.605688 13825 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.607280 13825 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:38.535779  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:38.498332 13831 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:38.539635  8578 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:38.505458 13845 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:38.546157  8576 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:38.613689 13845 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.614138 13845 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:38.550859  9327 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEARNER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:38.498664 13828 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:38.618930 13828 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.620508 13828 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:38.510854 13824 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:38.554352  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:38.558136  8575 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:38.564366  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:38.537751 13859 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.626672 13859 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.582906  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:38.586261 13405 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.629288 13859 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:38.626688  8579 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
W20260709 13:03:38.570709 13854 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.257s	user 0.001s	sys 0.000s
I20260709 13:03:38.533483 13242 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.568528 13876 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.570708 13866 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.570751 13676 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:38.682349  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 179525us
W20260709 13:03:38.684777  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 101955us
I20260709 13:03:38.574450 13858 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:38.687704  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:38.683954 13845 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.666985 13776 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:38.720739  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 116407us
I20260709 13:03:38.722571  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.726083  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.727634  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.673319 13681 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:38.667644 13864 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:38.679348 13852 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.337s	user 0.000s	sys 0.001s
I20260709 13:03:38.741755  8729 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:38.774353  8729 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:03:38.768479  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:38.686563 13846 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
W20260709 13:03:38.743350  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 55742.5us
I20260709 13:03:38.776350  9330 raft_consensus.cc:2410] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:03:38.776541  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:38.683063 13723 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:38.560592 13739 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.783345  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:38.684617 13830 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:38.791966  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.792837  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2487) took 1409 ms. Trace:
I20260709 13:03:38.805866  8727 rpcz_store.cc:276] 0709 13:03:37.383026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.387992 (+  4966us) service_pool.cc:224] Handling call
0709 13:03:37.388375 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.401848 (+ 13473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:37.401899 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:37.409986 (+  8087us) log.cc:844] Serialized 26 byte log entry
0709 13:03:37.901479 (+491493us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:37.901521 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.901595 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:38.792611 (+891016us) raft_consensus.cc:1685] finished
0709 13:03:38.792635 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:38.792778 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69968,"raft.queue_time_us":17602,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":13161,"thread_start_us":414014,"threads_started":1,"wal-append.queue_time_us":718203}
I20260709 13:03:38.807868  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.799129  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:38.801963  8730 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:38.804071  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 376014us
I20260709 13:03:38.641158 13842 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.769196 13776 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:38.822381  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:38.828862  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104453us
I20260709 13:03:38.772778 13767 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:38.832260  9330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:38.803185 13777 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:38.862112  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.715096 13676 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:38.677999 13879 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:38.829650 13784 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:38.833300 13830 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:38.835889 13764 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:38.880518  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:38.872298  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:38.883977  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 61720.1us
I20260709 13:03:38.810179 13833 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:38.920626  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:38.810109 13703 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:38.883795  8732 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:38.731426 13858 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:38.949139  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 76959.3us
I20260709 13:03:38.955817  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 955) took 1567 ms. Trace:
I20260709 13:03:38.686571 13876 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:38.899783  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:38.901566  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 214473us
W20260709 13:03:38.903098  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 102383us
I20260709 13:03:38.792174 13692 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:38.862018 13640 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:38.896694 13829 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.960s	user 0.013s	sys 0.001s
I20260709 13:03:38.944087  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:38.994161  8579 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:38.892158 13742 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:38.961992  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 129091us
I20260709 13:03:38.681288 13866 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:38.956387  8733 rpcz_store.cc:276] 0709 13:03:37.388570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.388723 (+   153us) service_pool.cc:224] Handling call
0709 13:03:37.389127 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.401058 (+ 11931us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:37.401094 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:37.401598 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 13:03:37.897155 (+495557us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:37.897217 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.897329 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:38.955514 (+1058185us) raft_consensus.cc:1685] finished
0709 13:03:38.955543 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:38.955739 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13524,"raft.queue_time_us":16709,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":12012,"thread_start_us":464337,"threads_started":1,"wal-append.queue_time_us":803846}
I20260709 13:03:39.022532  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:38.861683 13743 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
W20260709 13:03:38.947389 13870 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.435s	user 0.000s	sys 0.001s
W20260709 13:03:38.942615 13846 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:03:38.946000 13862 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:39.002828  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.044378  8730 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:39.000897  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:38.830255 13882 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.186s
I20260709 13:03:38.937899 13497 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:38.947247 13767 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:38.963332 13585 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:38.941913 13772 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.043948 13862 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:39.022221 13847 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:03:39.052286  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82089.9us
I20260709 13:03:39.072834 13898 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:38.947671 13703 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.073261 13829 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:39.072505 13868 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 13:03:39.083657 12171 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:39.068315  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:38.964529 13908 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:39.107437 13585 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:38.989095 13831 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:39.109625 13892 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.205s	user 0.000s	sys 0.003s
W20260709 13:03:39.111029 13895 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.174s	user 0.003s	sys 0.000s
I20260709 13:03:39.110208 13906 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.991456 13830 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:39.160625 13830 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:39.161230 13830 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:39.165946 13907 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 13:03:39.155670 13640 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:39.147225 13777 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:39.171550 13777 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.173722 13829 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:39.174294 13829 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:39.175325 13829 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:39.179998 13901 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.084s	user 0.002s	sys 0.000s
I20260709 13:03:39.182262 13712 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:39.187047 13876 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.187788 13832 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.188640 13842 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.117035  8733 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:03:39.115825 13405 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.060283 13706 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:39.026928 13848 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.133998 12171 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:39.139340  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 72207.7us
I20260709 13:03:39.221167 12171 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:38.917433 13879 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.152432 13845 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.153132 13916 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.153973 13906 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.032909 13742 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.192399  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:39.192732  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:39.193202  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:39.250725  9328 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:03:39.112569 13862 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:39.196825  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.146730 13899 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:39.045065 13824 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:39.202114 13806 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.020s	user 0.013s	sys 0.001s
W20260709 13:03:39.210367 13896 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 13:03:39.211428 13706 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.110971 13767 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.213737 13703 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.217649 13640 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:39.218309 13850 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.894s	user 0.003s	sys 0.001s
I20260709 13:03:39.222165 12171 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.224110 13585 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:39.277295 13886 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.150s	user 0.001s	sys 0.001s
I20260709 13:03:39.277374 13683 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:39.278108 13683 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:39.279158 13831 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:39.225201 13825 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.617s	user 0.007s	sys 0.006s
I20260709 13:03:39.123399 13772 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:39.159713 13756 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.584s	user 0.011s	sys 0.003s
I20260709 13:03:39.230825 13879 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.142967 13908 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.231842 13916 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.244956 13777 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.251524 13742 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.187810 13876 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.258219  8725 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:39.258253  8658 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:39.261740 13824 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:39.226948 13676 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.263820 13899 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.135588 13898 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.283833  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 952) took 1930 ms. Trace:
I20260709 13:03:39.300329 13906 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:39.301232 13888 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.236s	user 0.004s	sys 0.000s
I20260709 13:03:39.305037  8653 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:39.286089 13828 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.665s	user 0.015s	sys 0.000s
W20260709 13:03:39.282949 13802 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.277s	user 0.014s	sys 0.000s
I20260709 13:03:39.310079 13876 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:39.315966  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 342356us
I20260709 13:03:39.316831  8728 rpcz_store.cc:276] 0709 13:03:37.353799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.354377 (+   578us) service_pool.cc:224] Handling call
0709 13:03:37.354822 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.355261 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:37.355304 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:37.387196 (+ 31892us) log.cc:844] Serialized 26 byte log entry
0709 13:03:37.905278 (+518082us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:37.905341 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.905480 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:39.217551 (+1312071us) spinlock_profiling.cc:243] Waited 6.1 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:39.280185 (+ 62634us) raft_consensus.cc:1685] finished
0709 13:03:39.280214 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.283736 (+  3522us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2745,"raft.queue_time_us":20744,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":3432,"spinlock_wait_cycles":14033280,"thread_start_us":469495,"threads_started":1,"wal-append.queue_time_us":774394}
W20260709 13:03:39.328832 13633 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.762s	user 0.007s	sys 0.004s
W20260709 13:03:39.330293 13864 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.533s	user 0.008s	sys 0.004s
I20260709 13:03:39.308499 13756 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.331845 13706 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:39.338650 13633 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.303294 13772 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.346341  8655 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:39.359428  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1207) took 1008 ms. Trace:
I20260709 13:03:39.306190 13908 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:39.324167 13428 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.811s	user 0.007s	sys 0.005s
I20260709 13:03:39.280642 13640 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.360093  9324 rpcz_store.cc:276] 0709 13:03:38.350917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.351034 (+   117us) service_pool.cc:224] Handling call
0709 13:03:38.366672 (+ 15638us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:38.516041 (+149369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.516090 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:38.516661 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 13:03:38.720871 (+204210us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:38.720934 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.252100 (+531166us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b540001e354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:39.252149 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:39.352927 (+100778us) raft_consensus.cc:1685] finished
0709 13:03:39.352948 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.359352 (+  6404us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3685,"raft.queue_time_us":20851,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":23934,"spinlock_wait_cycles":333962112,"thread_start_us":152349,"threads_started":1,"wal-append.queue_time_us":316485}
I20260709 13:03:39.363277  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:39.364394 13899 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.321432 13242 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.368985 13428 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:39.352826 13681 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.000s
I20260709 13:03:39.279302 13683 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.332237 13858 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.378620 13633 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.383071 13428 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.386473 13683 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:39.389546  8577 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:39.391726  9326 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:39.357380 13864 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.395780 13864 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.397360 13765 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:39.360519 13906 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.409334  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:39.410164  8580 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:39.410293  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:39.410956  8576 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:39.416059 13706 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:39.416481  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:39.417167  8582 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:39.421213  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:39.422363  8584 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:39.441358  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.442210 13912 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.442834  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:39.444396  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:39.406190 13822 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.000s	user 0.011s	sys 0.004s
I20260709 13:03:39.405257 13400 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.443189  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.405694 13848 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.454468 12171 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:39.465242  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2691) took 1278 ms. Trace:
I20260709 13:03:39.465373 12171 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:39.465895 12171 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:39.465749  8658 rpcz_store.cc:276] 0709 13:03:38.187120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.477265 (+290145us) service_pool.cc:224] Handling call
0709 13:03:39.465052 (+987787us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b5400001754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:39.465168 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1594692480}
I20260709 13:03:39.463238  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2499) took 1074 ms. Trace:
I20260709 13:03:39.469769  8725 rpcz_store.cc:276] 0709 13:03:38.388634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.388814 (+   180us) service_pool.cc:224] Handling call
0709 13:03:38.389237 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.310133 (+920896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.463002 (+152869us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b540032df54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:39.463170 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1997964544}
I20260709 13:03:39.439910 13633 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.485703 13633 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.487560  8579 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
W20260709 13:03:39.446878 13888 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:03:39.482930 13703 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:39.411998 13906 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.449604 13898 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:39.443218 13705 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.463492 13640 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.476181 13400 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.496567 13633 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.449376 13831 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:39.507898 12171 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.511643 13906 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.486407 13756 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.485978 13866 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.461985 13876 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.453111 13822 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.545315  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 967) took 1133 ms. Trace:
I20260709 13:03:39.545852  8577 rpcz_store.cc:276] 0709 13:03:38.412232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.412372 (+   140us) service_pool.cc:224] Handling call
0709 13:03:39.545102 (+1132730us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b540017a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:39.545253 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2044556672}
I20260709 13:03:39.416640 13706 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.449744 13916 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.514210 13705 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:39.554467  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 584699us
I20260709 13:03:39.557118  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2489) took 2168 ms. Trace:
I20260709 13:03:39.563402  8582 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:39.492142 13703 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.485077 13908 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:39.547838  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.549223  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:39.551896 13898 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:39.452284 13912 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.525115  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:39.548444 13764 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.602s	user 0.005s	sys 0.010s
I20260709 13:03:39.543970 13830 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.557749  8724 rpcz_store.cc:276] 0709 13:03:37.385318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.385934 (+   616us) service_pool.cc:224] Handling call
0709 13:03:37.409443 (+ 23509us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:37.416684 (+  7241us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:37.593720 (+177036us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b18003f43c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:03:38.329554 (+735834us) log.cc:844] Serialized 14 byte log entry
0709 13:03:38.897194 (+567640us) log.cc:844] Serialized 14 byte log entry
0709 13:03:38.897391 (+   197us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:38.938352 (+ 40961us) log.cc:844] Serialized 52 byte log entry
0709 13:03:38.938640 (+   288us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:38.943338 (+  4698us) log.cc:844] Serialized 14 byte log entry
0709 13:03:39.211444 (+268106us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b18003f43c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:39.546289 (+334845us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b18003f43c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:03:39.553133 (+  6844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.553244 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:39.553312 (+    68us) raft_consensus.cc:1685] finished
0709 13:03:39.553337 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.553555 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":886022,"prepare.queue_time_us":972845,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":158,"raft.queue_time_us":263110,"raft.run_cpu_time_us":529,"raft.run_wall_time_us":50091,"spinlock_wait_cycles":36175232,"thread_start_us":1045404,"threads_started":2,"wal-append.queue_time_us":416753}
I20260709 13:03:39.566309 13904 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.565090  8576 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:39.572831 13772 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:39.573536  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 457238us
I20260709 13:03:39.571552 13703 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:39.579442 13784 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.642s	user 0.005s	sys 0.007s
W20260709 13:03:39.602561  8655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2940) took 1452 ms (client timeout 1930 ms). Trace:
W20260709 13:03:39.603140  8655 rpcz_store.cc:269] 0709 13:03:38.149699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.149823 (+   124us) service_pool.cc:224] Handling call
0709 13:03:39.602380 (+1452557us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:39.602490 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2577668480}
I20260709 13:03:39.542639 13831 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.586337 13885 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:39.571302 13516 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:39.587360 13908 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:39.579571 13776 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.708s	user 0.008s	sys 0.002s
I20260709 13:03:39.589056 13867 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:39.589872  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:39.608781  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 2932) took 2114 ms (client timeout 1714 ms). Trace:
W20260709 13:03:39.609267  8653 rpcz_store.cc:269] 0709 13:03:37.487791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.495264 (+  7473us) service_pool.cc:224] Handling call
0709 13:03:39.601752 (+2106488us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400470754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:39.601916 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3560121600}
I20260709 13:03:39.576810 13640 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:39.611217  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.611477  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:39.591244  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:39.591917  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:39.580266 13858 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.623158 13776 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.575290  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:39.628742  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 53545.2us
I20260709 13:03:39.612259  8728 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:39.577499 13741 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:03:39.595275  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.610791 13784 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:39.621044 13912 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:39.623129 12171 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.608160 13772 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.622670  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.636174  8733 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:39.621305  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.578274 13866 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:39.590389 13862 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.577332 13568 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 13:03:39.576982  8579 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:39.627005 13705 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.625335 13885 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.651283  8730 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:39.651535  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.631408 13867 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:39.584329 13897 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.633579  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.635212  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:39.656219  8728 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:39.637449  8724 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:39.637921 13908 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:39.626392 13516 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.628553 13640 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.655184  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.643085 13772 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.675469 12171 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:39.676044 12171 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:39.677363  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:39.678058  8576 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:39.637264 13858 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 13:03:39.676760  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.673800 13908 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:39.686427 13822 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.688752  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:39.689754  9326 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:39.694661  8725 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 13:03:39.695890  8576 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:39.696842  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:39.699828  8579 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:39.664670 13705 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:39.697386  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:39.688122 13830 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.694551 12171 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:39.687002 13908 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.588655 13904 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.708688 13912 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:39.711079 13862 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:39.694342 13879 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:39.715198 13858 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:39.684851 13897 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.723827  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.703593 13683 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:03:39.724689  8729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.029s	sys 0.109s
I20260709 13:03:39.719888 13912 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:39.735224  8729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.030s	sys 0.109s
I20260709 13:03:39.716382 13829 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:39.704186 13867 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.024s
W20260709 13:03:39.758546  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.010s	sys 0.025s
W20260709 13:03:39.759363  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.020s	sys 0.020s
W20260709 13:03:39.760213  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.020s	sys 0.021s
I20260709 13:03:39.737138  8579 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:39.742069  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:39.764550  8576 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:39.724749 13879 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.762709  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.718061 13706 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:39.718684 13705 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:39.759518  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.010s	sys 0.026s
I20260709 13:03:39.737015 13912 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.734918 13683 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:39.791375  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:39.799817  9323 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:39.801752  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:39.803270  8581 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.000s	sys 0.025s
W20260709 13:03:39.803722  8581 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.000s	sys 0.025s
I20260709 13:03:39.724424 13848 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:39.798310  9331 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.021s	sys 0.019s
W20260709 13:03:39.807941  9331 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.021s	sys 0.019s
W20260709 13:03:39.800384  8656 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.123s	user 0.061s	sys 0.064s
I20260709 13:03:39.804423  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:39.809638  8656 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.133s	user 0.061s	sys 0.064s
W20260709 13:03:39.796870  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.006s	sys 0.017s
W20260709 13:03:39.806865  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44236.3us
W20260709 13:03:39.811297  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.006s	sys 0.018s
W20260709 13:03:39.811667  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 926336us
I20260709 13:03:39.790086 13879 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:39.813515  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:39.792898 13776 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.735446 13858 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.806955  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:39.861459  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 54566us
I20260709 13:03:39.810895  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:39.865767  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54967.8us
W20260709 13:03:39.815271  8583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.143s	user 0.012s	sys 0.017s
W20260709 13:03:39.872370  8583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.200s	user 0.013s	sys 0.017s
I20260709 13:03:39.827003  9326 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:39.827816  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:39.824314  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:39.834290  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.875250  9327 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:39.876806  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.881217  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:39.820559  9323 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
I20260709 13:03:39.759088 13841 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:39.812742 13831 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.889078  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.890313 13772 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:39.824702 13856 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:39.892530  8728 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:39.893646  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.872843 13676 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:39.829267 13822 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:03:39.890774 13516 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.880407 13692 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:39.917659  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.898782  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.873237 13866 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.896888  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.879976 13876 raft_consensus.cc:2764] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:03:39.909214  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.823841 13848 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.930117  8728 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:39.906771  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:39.879015 13400 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:39.921432  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:39.936213  9322 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.108s	user 0.014s	sys 0.019s
W20260709 13:03:39.936591  9322 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.108s	user 0.014s	sys 0.019s
I20260709 13:03:39.936879  8725 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:39.922068  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:39.924437  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.015s	sys 0.038s
I20260709 13:03:39.884186 13916 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
W20260709 13:03:39.928427  8652 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.512s	user 0.014s	sys 0.028s
W20260709 13:03:39.943219  8652 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.528s	user 0.014s	sys 0.028s
W20260709 13:03:39.939980  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.223s	user 0.016s	sys 0.038s
I20260709 13:03:39.946688  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.944854  8548 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:39.907644 13749 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.957973  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.958963  8724 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:39.949802  8548 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:39.952385  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:39.960772  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:39.956216  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:39.963636  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:39.965960  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2941) took 1814 ms. Trace:
I20260709 13:03:39.954912 13328 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 13:03:39.969120 13884 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:39.969668 13884 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:39.970163 13884 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:39.910408 13841 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:39.917915 13712 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:39.972182  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:39.928014 13692 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:39.929955 13856 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:39.967042  8652 rpcz_store.cc:276] 0709 13:03:38.151144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.151295 (+   151us) service_pool.cc:224] Handling call
0709 13:03:38.185597 (+ 34302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:38.185826 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:38.414510 (+228684us) log.cc:844] Serialized 14 byte log entry
0709 13:03:39.965488 (+1550978us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:39.965560 (+    72us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:39.965620 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.965710 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.965881 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35396,"prepare.queue_time_us":392176,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":22507,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":1724556,"threads_started":2}
I20260709 13:03:39.971944  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:40.042419 13725 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:40.043224 13725 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:40.043586 13725 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:40.043933 13725 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.948764 13400 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:40.048731  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:39.972854  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:40.045655  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82121.1us
I20260709 13:03:40.050221  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:40.051020  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:40.051311  8650 raft_consensus.cc:2410] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:03:40.051709  9329 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:40.047557  9328 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:03:40.053340  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:39.948175 13706 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:40.048372  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:39.982479  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:39.966042 13864 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:40.057830  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:39.967608 13907 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:39.896608 13683 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:40.061849  8549 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.572s (SENT) [suppressed 51 similar messages]
I20260709 13:03:40.079895  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:40.084455  9328 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:39.949826 13516 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:39.918913 13676 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:39.942065 13866 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.036671 13692 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:40.106468  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:40.108215  9328 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:40.053850 13712 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.086647 13916 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:39.926961 13705 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.049576 13640 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.048141 13831 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.048501 13617 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:40.059116 13859 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:39.967392 13749 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.081719 13706 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:40.092389 13739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.954s	user 0.027s	sys 0.027s
I20260709 13:03:40.128129  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:40.082131 13400 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.132087  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:40.059998 13751 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:03:40.181650  8731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.166s	user 0.021s	sys 0.049s
W20260709 13:03:40.183416  8731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.168s	user 0.021s	sys 0.049s
I20260709 13:03:40.104626 13830 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:40.120970 13864 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:40.127036  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:40.196054  9328 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:40.174026 13749 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.175678 13676 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.178310  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:40.201293  8655 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.104416 13907 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.153954 13516 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:40.136899  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.192973  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:40.181440 13739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.043s	user 0.027s	sys 0.027s
I20260709 13:03:40.164057 13856 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.202221  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:40.227771  8658 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:40.203949  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:40.228772  8650 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:40.185530 13751 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:40.230037  8655 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:40.188894 13830 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:40.197729  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:40.208582  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 920) took 1617 ms. Trace:
I20260709 13:03:40.178058 13617 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:40.181792 13915 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:40.226593  8726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.024s	sys 0.039s
I20260709 13:03:40.239058  9331 rpcz_store.cc:276] 0709 13:03:38.591065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.591220 (+   155us) service_pool.cc:224] Handling call
0709 13:03:38.591727 (+   507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:38.873874 (+282147us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.874012 (+   138us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:38.901141 (+ 27129us) log.cc:844] Serialized 26 byte log entry
0709 13:03:39.808372 (+907231us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:39.808414 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.808479 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:40.208302 (+399823us) raft_consensus.cc:1685] finished
0709 13:03:40.208331 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.208509 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43320,"raft.queue_time_us":242790,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":18852,"thread_start_us":1107821,"threads_started":2,"wal-append.queue_time_us":1243010}
W20260709 13:03:40.248303  8726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.024s	sys 0.039s
I20260709 13:03:40.167481 13831 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.181501 13859 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.257256  8650 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:40.258370  8575 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.312s	user 0.024s	sys 0.044s
W20260709 13:03:40.262182  8575 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.315s	user 0.024s	sys 0.044s
W20260709 13:03:40.231855 13767 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.921s	user 0.015s	sys 0.060s
W20260709 13:03:40.246732 13780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.011s	sys 0.023s
W20260709 13:03:40.257990  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.012s	sys 0.022s
W20260709 13:03:40.267066  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.012s	sys 0.022s
I20260709 13:03:40.189908 13400 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:40.269891  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 714569us
I20260709 13:03:40.245736 13830 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.228226 13904 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.165596 13916 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:40.253242 13428 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.006s	sys 0.029s
I20260709 13:03:40.236004 13751 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:40.272895  8658 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:40.242400 13867 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:40.296607  9325 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.442s	user 0.015s	sys 0.054s
W20260709 13:03:40.297057  9325 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.466s	user 0.015s	sys 0.054s
W20260709 13:03:40.266623 13780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.012s	sys 0.023s
W20260709 13:03:40.282253  8659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.302s	user 0.006s	sys 0.088s
W20260709 13:03:40.263278 13767 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.016s	sys 0.061s
W20260709 13:03:40.288930  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 91269.7us
W20260709 13:03:40.223238 13585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.016s	sys 0.035s
W20260709 13:03:40.300081  8659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.320s	user 0.007s	sys 0.088s
W20260709 13:03:40.278630 13914 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.117s	user 0.001s	sys 0.001s
W20260709 13:03:40.303431  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.514s	user 0.018s	sys 0.014s
W20260709 13:03:40.295727 13428 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.006s	sys 0.029s
I20260709 13:03:40.259959 13915 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.186770 13841 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:40.307241 13585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.027s	user 0.016s	sys 0.036s
I20260709 13:03:40.301761 13848 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:40.354120 13765 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.956s	user 0.014s	sys 0.000s
I20260709 13:03:40.337030 13400 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.335561 13866 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.306422 13867 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.367877 13712 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.370748 13694 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.386052 13694 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:40.387790 13829 consensus_meta.cc:287] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.628s	user 0.007s	sys 0.003s
I20260709 13:03:40.388752 13694 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.382447 13884 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.371428 13897 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.383839 13915 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.378464 13925 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:40.384538 13828 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:40.380198 13848 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:40.393663 13639 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.016s	sys 0.030s
I20260709 13:03:40.390566 13833 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.401662 13706 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.398425 13925 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:40.418197 13884 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.412703 13725 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.421952 13859 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.423885 13725 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.304024 13864 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.411413 13831 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.412485 13925 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:40.428534 13858 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.409948 13833 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.407174 13848 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.451501 13885 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:40.467618 13912 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.647s	user 0.004s	sys 0.008s
I20260709 13:03:40.479112 13912 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:40.476106 13926 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 13:03:40.469096 13879 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.596s	user 0.008s	sys 0.004s
I20260709 13:03:40.482331 13879 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.483166 13907 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.465268 13916 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.484930 13867 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.487313 13879 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.488965 13912 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:40.534430  8578 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.009s	sys 0.028s
W20260709 13:03:40.535389  8578 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.009s	sys 0.028s
I20260709 13:03:40.543999  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2311) took 2011 ms. Trace:
I20260709 13:03:40.544971  8583 rpcz_store.cc:276] 0709 13:03:38.532415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.534829 (+  2414us) service_pool.cc:224] Handling call
0709 13:03:38.554061 (+ 19232us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:38.561085 (+  7024us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.561118 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:38.567943 (+  6825us) log.cc:844] Serialized 26 byte log entry
0709 13:03:39.872822 (+1304879us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:39.872877 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.872944 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:40.543057 (+670113us) raft_consensus.cc:1685] finished
0709 13:03:40.543127 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.543701 (+   574us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6541,"raft.queue_time_us":9964,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":7195,"thread_start_us":1200028,"threads_started":1,"wal-append.queue_time_us":1674559}
W20260709 13:03:40.552548  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.763s	user 0.019s	sys 0.014s
W20260709 13:03:40.553175  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:40.553051  8656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.005s	sys 0.001s
W20260709 13:03:40.557865  8656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.006s	sys 0.001s
I20260709 13:03:40.569440  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2761) took 1847 ms. Trace:
I20260709 13:03:40.570430  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 959) took 1847 ms. Trace:
I20260709 13:03:40.570075  8731 rpcz_store.cc:276] 0709 13:03:38.721337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.724906 (+  3569us) service_pool.cc:224] Handling call
0709 13:03:38.725327 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:38.898508 (+173181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.898551 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:38.917002 (+ 18451us) log.cc:844] Serialized 26 byte log entry
0709 13:03:39.212979 (+295977us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b480018c258. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:40.186486 (+973507us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:40.186546 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.188149 (+  1603us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:40.568463 (+380314us) raft_consensus.cc:1685] finished
0709 13:03:40.568505 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.569082 (+   577us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":523347,"raft.queue_time_us":175362,"raft.run_cpu_time_us":1175,"raft.run_wall_time_us":296099,"spinlock_wait_cycles":45855488,"thread_start_us":1171230,"threads_started":1,"wal-append.queue_time_us":1477262}
W20260709 13:03:40.573679  8733 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.002s	sys 0.058s
W20260709 13:03:40.574204  8733 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.002s	sys 0.058s
I20260709 13:03:40.571045  8729 rpcz_store.cc:276] 0709 13:03:38.723009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.775437 (+ 52428us) service_pool.cc:224] Handling call
0709 13:03:38.825388 (+ 49951us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:38.950923 (+125535us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.950969 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:38.951532 (+   563us) log.cc:844] Serialized 26 byte log entry
0709 13:03:39.737619 (+786087us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:39.737677 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.798044 (+ 60367us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54000eae54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:39.798113 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:40.569690 (+771577us) raft_consensus.cc:1685] finished
0709 13:03:40.569730 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.570224 (+   494us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287767808,"thread_start_us":785712,"threads_started":1,"wal-append.queue_time_us":1168294}
I20260709 13:03:40.584661  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:40.584591  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:40.586324  8731 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:40.591460  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 921) took 1999 ms. Trace:
I20260709 13:03:40.591995  9322 rpcz_store.cc:276] 0709 13:03:38.592398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.592549 (+   151us) service_pool.cc:224] Handling call
0709 13:03:38.592883 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:38.769643 (+176760us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.769681 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:38.781624 (+ 11943us) log.cc:844] Serialized 52 byte log entry
0709 13:03:39.936988 (+1155364us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:39.937030 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.937106 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:40.590814 (+653708us) raft_consensus.cc:1685] finished
0709 13:03:40.590848 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.591215 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50200,"raft.queue_time_us":286536,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":11358,"thread_start_us":1284948,"threads_started":2,"wal-append.queue_time_us":1541267}
I20260709 13:03:40.596032  9326 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:40.596184  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:40.596208  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:40.597420  9328 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:40.597558  9331 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:40.599293  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:40.603644  8654 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:40.311800 13516 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:40.611882 12171 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.008s	sys 0.032s
W20260709 13:03:40.612521 12171 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.008s	sys 0.032s
I20260709 13:03:40.612403 13712 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:40.613725 13712 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.372272 13617 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:40.405375 13639 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.016s	sys 0.030s
I20260709 13:03:40.618057 13683 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:40.622907 13727 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:40.623901 13727 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.627141 13723 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.617381 13706 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.628105 13723 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.628557 13706 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.631867 13727 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.632607 13706 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:40.634158 13706 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:40.635119 13706 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:40.636749  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.642037 13639 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:40.643517 13703 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.035s	user 0.016s	sys 0.005s
I20260709 13:03:40.643206 13639 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.619241 13712 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.647747  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:40.650878 13723 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.652319 13328 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:40.653146 13676 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:40.661850 13739 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:40.661965 13824 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.661007 13706 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:40.741627  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.750309  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:40.747516 13703 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.792773 13912 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:40.797972 13867 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.816083  8657 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:40.819036 13833 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.820569  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:40.840099  8657 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:40.798409  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 151031us
I20260709 13:03:40.814894 13723 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:40.844847 13723 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.848508 13723 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:40.848623 13712 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:40.849861  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:40.811725 13328 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.865337 13328 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:40.858091  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:40.866628  8657 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:40.849373 13712 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:40.858194 13858 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:40.809581 13739 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.822077 13824 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.887562  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:40.894804  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:40.798518 13838 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.133s
I20260709 13:03:40.887384 13712 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.841614 13676 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:40.886922 13858 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:40.908154  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:40.909014  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 57075.8us
I20260709 13:03:40.911410  8657 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.874022 13912 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:40.927881  8581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.000s	user 0.009s	sys 0.031s
W20260709 13:03:40.928484  8581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.009s	sys 0.031s
I20260709 13:03:40.907902 13858 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:40.891062 13867 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:40.948189  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:40.967383  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:40.968752  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:40.988554  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:40.993238  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:40.974660 13328 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:41.002265  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.005425  8727 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
W20260709 13:03:41.012073  9327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.000s	sys 0.036s
W20260709 13:03:41.012637  9327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.000s	sys 0.037s
I20260709 13:03:41.012813  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2332) took 1474 ms. Trace:
I20260709 13:03:41.013283  8578 rpcz_store.cc:276] 0709 13:03:39.538477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.538788 (+   311us) service_pool.cc:224] Handling call
0709 13:03:39.539220 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.539452 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:40.536062 (+996610us) log.cc:844] Serialized 14 byte log entry
0709 13:03:41.005157 (+469095us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:41.012244 (+  7087us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:41.012364 (+   120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.012487 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.012748 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1462481,"raft.queue_time_us":9963,"thread_start_us":1431842,"threads_started":2}
W20260709 13:03:41.016522  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.017508 13908 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.014s	sys 0.045s
W20260709 13:03:41.017973 13908 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.014s	sys 0.045s
I20260709 13:03:41.020352  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:41.021337  8724 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:41.029740  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.007s	sys 0.021s
W20260709 13:03:41.030335  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.007s	sys 0.021s
W20260709 13:03:41.030805  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 727170us
W20260709 13:03:41.033567  8575 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.012s	sys 0.021s
W20260709 13:03:41.034245  8575 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.012s	sys 0.022s
I20260709 13:03:41.033725  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 970) took 1328 ms. Trace:
W20260709 13:03:41.035463 13428 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.021s	sys 0.008s
W20260709 13:03:41.036103 13428 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.022s	sys 0.008s
I20260709 13:03:41.037818 13938 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:41.038738 13938 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.043426  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:41.046926  9328 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:03:41.048111  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.050282 13937 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:41.051059 13937 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:41.051569 13937 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:41.054067  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.055018  8731 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:41.055881  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.056610  8731 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:03:41.057636 13864 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.011s	sys 0.015s
W20260709 13:03:41.058063 13864 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.011s	sys 0.015s
I20260709 13:03:41.061342 12171 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:41.061507  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.062160 12171 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.063687  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:41.064878 12171 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:41.065264 13764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.296s	user 0.018s	sys 0.037s
I20260709 13:03:41.064885  8726 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:41.011083 13328 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:41.066251 13328 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:41.066841  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:41.067077  9325 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.024s	sys 0.007s
I20260709 13:03:41.067449  8726 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:41.068531  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:41.069552  8658 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.021s	sys 0.011s
W20260709 13:03:41.070039  8658 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.021s	sys 0.011s
I20260709 13:03:41.070840  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:41.070011  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.072074  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:41.072679  8727 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:41.073529 13756 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:41.073977  8728 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:41.074965  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 838) took 2892 ms. Trace:
I20260709 13:03:41.010870  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:41.075764  8730 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.989774 13703 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.077767  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:41.067620  9325 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.024s	sys 0.007s
I20260709 13:03:41.089234  8729 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 13:03:41.090492  8724 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
W20260709 13:03:41.065902 13764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.411s	user 0.018s	sys 0.037s
I20260709 13:03:41.071913 13848 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.095068  9331 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:03:41.035511  8733 rpcz_store.cc:276] 0709 13:03:39.705107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.705389 (+   282us) service_pool.cc:224] Handling call
0709 13:03:39.725456 (+ 20067us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.725720 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:40.574913 (+849193us) log.cc:844] Serialized 14 byte log entry
0709 13:03:40.981636 (+406723us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:40.994544 (+ 12908us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:40.998712 (+  4168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.011265 (+ 12553us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.033667 (+ 22402us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50669,"prepare.queue_time_us":1269726,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":26987,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":37059,"thread_start_us":1186938,"threads_started":2}
W20260709 13:03:41.101468 13340 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.065s	user 0.021s	sys 0.043s
W20260709 13:03:41.102111 13340 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.223s	user 0.021s	sys 0.043s
I20260709 13:03:41.103616  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.105319  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2690) took 2918 ms. Trace:
I20260709 13:03:41.105988  8656 rpcz_store.cc:276] 0709 13:03:38.186799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.603197 (+416398us) service_pool.cc:224] Handling call
0709 13:03:38.603515 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:39.823841 (+1220326us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:39.823903 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:39.873134 (+ 49231us) log.cc:844] Serialized 26 byte log entry
0709 13:03:40.558554 (+685420us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:40.558673 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.558835 (+   162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.104973 (+546138us) raft_consensus.cc:1685] finished
0709 13:03:41.105008 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.105219 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77794,"raft.queue_time_us":116664,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5480,"thread_start_us":1763841,"threads_started":2,"wal-append.queue_time_us":1200435}
I20260709 13:03:41.108922  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.112450  8656 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:03:41.113934  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.115190  8656 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:41.116462  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:41.118237  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 85641.2us
W20260709 13:03:41.069597 13705 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.007s	sys 0.020s
I20260709 13:03:41.130584  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.074640 13756 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.091617 13639 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:41.147192  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:41.137106 13705 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.008s	sys 0.020s
I20260709 13:03:41.152747  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:41.148149  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57146us
I20260709 13:03:41.138072 13908 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.170642  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.167367  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:41.195683  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.196477  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 49506.1us
W20260709 13:03:41.203626  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.203017  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.213485  8654 raft_consensus.cc:2393] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:41.192297 13692 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.862s	user 0.014s	sys 0.011s
W20260709 13:03:41.219113  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49127.1us
W20260709 13:03:41.218066  8650 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.012s	sys 0.040s
W20260709 13:03:41.229030  8651 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.008s	sys 0.033s
W20260709 13:03:41.222724  8549 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.096s	user 0.018s	sys 0.048s
W20260709 13:03:41.255287  8650 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.965s	user 0.013s	sys 0.041s
I20260709 13:03:41.261353  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:41.262292 13867 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:41.264602 13617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.015s	sys 0.017s
I20260709 13:03:41.264381 13939 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.174s
I20260709 13:03:41.265565 13908 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.266253 13640 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:41.267170 13772 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:41.267135 13640 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.267957  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.268409 13908 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.268689  9330 raft_consensus.cc:606] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Received request to transfer leadership to TS 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:41.269006  8654 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:41.249068 12171 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.269236  9330 raft_consensus.cc:614] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:03:41.269716  8549 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.186s	user 0.019s	sys 0.048s
W20260709 13:03:41.265142 13617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.015s	sys 0.018s
W20260709 13:03:41.272011  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 66465.4us
W20260709 13:03:41.288081  8651 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.089s	user 0.009s	sys 0.034s
I20260709 13:03:41.233178 13764 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:03:41.305174  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.140s	user 0.029s	sys 0.037s
W20260709 13:03:41.305738  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.258s	user 0.029s	sys 0.037s
W20260709 13:03:41.306274  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 1.33425e+06us
I20260709 13:03:41.267982 13772 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:41.310622  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.310755 13772 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.312067 13908 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.311630  8733 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:41.321564  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2319) took 2746 ms. Trace:
I20260709 13:03:41.325802  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:41.305174  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.218s	user 0.008s	sys 0.021s
I20260709 13:03:41.075394  8659 rpcz_store.cc:276] 0709 13:03:38.182932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.598374 (+415442us) service_pool.cc:224] Handling call
0709 13:03:38.598820 (+   446us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:38.884626 (+285806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.884675 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:38.916903 (+ 32228us) log.cc:844] Serialized 52 byte log entry
0709 13:03:39.248861 (+331958us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b48001b2358. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:40.304519 (+1055658us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:41.059036 (+754517us) log.cc:844] Serialized 14 byte log entry
0709 13:03:41.061176 (+  2140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.074445 (+ 13269us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.074527 (+    82us) raft_consensus.cc:1685] finished
0709 13:03:41.074622 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.074762 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77288,"prepare.queue_time_us":489749,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":35,"raft.queue_time_us":80211,"raft.run_cpu_time_us":1602,"raft.run_wall_time_us":567241,"spinlock_wait_cycles":5642624,"thread_start_us":1932994,"threads_started":3,"wal-append.queue_time_us":1516090}
I20260709 13:03:41.312844 13908 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:41.320977  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:41.340272  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.301522 13764 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:41.295758 13639 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
W20260709 13:03:41.345682  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 1.28372e+06us
W20260709 13:03:41.345947 13841 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.012s	sys 0.017s
I20260709 13:03:41.319782 13705 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.365067  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.342684  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:41.345206  8655 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:41.357481  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.395s	user 0.008s	sys 0.024s
I20260709 13:03:41.266656 13848 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.086s
I20260709 13:03:41.271194 13692 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:41.367673 12171 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:41.369539  8654 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:41.375759  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:41.371150  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 930) took 1399 ms. Trace:
I20260709 13:03:41.370265 13908 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.370517  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39404 (request call id 21) took 2556 ms. Trace:
W20260709 13:03:41.370761 13841 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.012s	sys 0.017s
I20260709 13:03:41.325700  8581 rpcz_store.cc:276] 0709 13:03:38.553097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.610239 (+ 57142us) service_pool.cc:224] Handling call
0709 13:03:38.610643 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:39.865520 (+1254877us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.865552 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:39.883745 (+ 18193us) log.cc:844] Serialized 26 byte log entry
0709 13:03:40.929305 (+1045560us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:40.929466 (+   161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.135233 (+205767us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54002cabd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:41.135308 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.269946 (+134638us) raft_consensus.cc:1685] finished
0709 13:03:41.269967 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.299825 (+ 29858us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":144387,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":967669120,"thread_start_us":1781864,"threads_started":2,"wal-append.queue_time_us":1325825}
I20260709 13:03:41.380131  9330 rpcz_store.cc:276] 0709 13:03:38.814038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.832113 (+ 18075us) service_pool.cc:224] Handling call
0709 13:03:41.370156 (+2538043us) spinlock_profiling.cc:243] Waited 2.4 s on lock 0x7b5400542754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c10853 00007f140ebde60a 00007f14081647d4 00007f1408164762 00007f1408164691 00007f140816460c 00007f14081638c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:41.370332 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5509939072}
I20260709 13:03:41.387483 13705 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:41.390107  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 47558.1us
I20260709 13:03:41.391579 13705 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:41.391736 13947 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 13:03:41.393545 12171 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.393940 13676 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:41.395804 13925 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.944s	user 0.012s	sys 0.002s
I20260709 13:03:41.395910 13712 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.396451 13925 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.398455 13867 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:41.399030 13867 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:41.399498 13867 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:41.401522  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 1.4407e+06us
I20260709 13:03:41.402870  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:41.407454 13876 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:41.409521 13692 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.409854  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 970) took 2516 ms. Trace:
I20260709 13:03:41.410365  8575 rpcz_store.cc:276] 0709 13:03:38.893202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.893349 (+   147us) service_pool.cc:224] Handling call
0709 13:03:38.942031 (+ 48682us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:39.204172 (+262141us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b48002dffd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:40.263513 (+1059341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:40.263548 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:40.264122 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 13:03:41.034948 (+770826us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:41.035047 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.035203 (+   156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.409353 (+374150us) raft_consensus.cc:1685] finished
0709 13:03:41.409385 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.409753 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19850,"raft.queue_time_us":163262,"raft.run_cpu_time_us":1792,"raft.run_wall_time_us":91600,"spinlock_wait_cycles":13162240,"thread_start_us":2065709,"threads_started":2,"wal-append.queue_time_us":1126898}
I20260709 13:03:41.412770  8652 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:41.414599  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2951) took 1293 ms. Trace:
I20260709 13:03:41.415040  8652 rpcz_store.cc:276] 0709 13:03:40.121176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.219647 (+ 98471us) service_pool.cc:224] Handling call
0709 13:03:40.245060 (+ 25413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:41.413311 (+1168251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.414217 (+   906us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001310d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:41.414509 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2684515072}
I20260709 13:03:41.416474  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:41.418184  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 195975us
I20260709 13:03:41.418720 13930 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.419557 13930 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.420480 13908 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:41.420995 13908 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:41.421455 13908 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:41.422269  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.423107  8656 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:41.424218  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 919) took 2837 ms. Trace:
I20260709 13:03:41.424634  8656 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:41.439759 13912 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:41.442767 13751 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.136s	user 0.008s	sys 0.011s
I20260709 13:03:41.443375 13751 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.443390  8659 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:41.446183 13751 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.448863 13751 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:41.448684 13937 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.449822 13946 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:41.454330  8656 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:41.456480 13925 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.456674 13824 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.459852 13640 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.460507  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:41.463590 13756 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.463433 13328 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.441480 13876 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:41.465586  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:41.461052  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:41.468065  8730 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:41.376839  9327 rpcz_store.cc:276] 0709 13:03:39.971561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.971740 (+   179us) service_pool.cc:224] Handling call
0709 13:03:40.203584 (+231844us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:40.203823 (+   239us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:40.204134 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:40.204175 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:40.204807 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 13:03:41.013228 (+808421us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:41.013325 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.013585 (+   260us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.370787 (+357202us) raft_consensus.cc:1685] finished
0709 13:03:41.370813 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.371068 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5827328,"thread_start_us":758684,"threads_started":1,"wal-append.queue_time_us":1139193}
I20260709 13:03:41.493234  8725 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:41.481858  8727 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:41.485731 13739 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:41.524063  8733 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:41.532680  8726 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:41.424659  9325 rpcz_store.cc:276] 0709 13:03:38.587003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.587187 (+   184us) service_pool.cc:224] Handling call
0709 13:03:38.587531 (+   344us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:38.587989 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:38.588372 (+   383us) log.cc:844] Serialized 14 byte log entry
0709 13:03:39.179456 (+591084us) spinlock_profiling.cc:243] Waited 5.54 ms on lock 0x7b24000c54c8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:03:40.297432 (+1117976us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:40.329747 (+ 32315us) log.cc:844] Serialized 52 byte log entry
0709 13:03:41.206724 (+876977us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:41.220193 (+ 13469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.227426 (+  7233us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.423860 (+196434us) raft_consensus.cc:1685] finished
0709 13:03:41.423953 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.424136 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15405,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":36,"raft.queue_time_us":362162,"raft.run_cpu_time_us":433,"raft.run_wall_time_us":59954,"spinlock_wait_cycles":12739968,"thread_start_us":2464819,"threads_started":3,"wal-append.queue_time_us":1479702,"wal-append.run_cpu_time_us":5362,"wal-append.run_wall_time_us":1180552}
I20260709 13:03:41.537388  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.542493  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.543984  8729 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:03:41.555110  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.555943  8727 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:41.555991  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.556883  8731 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:41.531958 13756 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:41.557309  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 388037us
I20260709 13:03:41.488504  8659 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
W20260709 13:03:41.545634 13830 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.248s	user 0.007s	sys 0.011s
I20260709 13:03:41.522200 13723 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:41.564481  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 104206us
I20260709 13:03:41.521929 13838 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:41.546459 13824 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:41.574848  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 113952us
I20260709 13:03:41.576129  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:41.578043  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.580063  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:41.582108  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:41.582676 13930 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.583024  9331 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:41.586129  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:41.586702  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51652.1us
I20260709 13:03:41.588114  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:41.535512 13751 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:41.604842  8652 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:41.554764 13876 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:41.609091  8656 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:41.611407  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:41.613242  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2957) took 1426 ms. Trace:
I20260709 13:03:41.613876 13723 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.550084 13946 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.615417 13838 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.616590 13824 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.583472 13930 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:03:41.618728  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.620792  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:41.622308  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:41.623432  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:41.624701 13856 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.025s	user 0.018s	sys 0.027s
I20260709 13:03:41.631268 13908 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.632089  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.644213  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.634047  8658 rpcz_store.cc:276] 0709 13:03:40.187457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.299003 (+111546us) service_pool.cc:224] Handling call
0709 13:03:40.299515 (+   512us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:41.070619 (+771104us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:41.070667 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:41.071432 (+   765us) log.cc:844] Serialized 52 byte log entry
0709 13:03:41.373757 (+302325us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:41.465853 (+ 92096us) log.cc:844] Serialized 14 byte log entry
0709 13:03:41.485592 (+ 19739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.485714 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.599129 (+113415us) raft_consensus.cc:1685] finished
0709 13:03:41.599149 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.613176 (+ 14027us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24430,"prepare.queue_time_us":68840,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":99144,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":333,"thread_start_us":1065542,"threads_started":2,"wal-append.queue_time_us":469003}
I20260709 13:03:41.710733  8725 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
W20260709 13:03:41.739550  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.892333  8730 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:41.564276 13830 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:41.660377  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:41.901090  8657 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.977s	user 0.013s	sys 0.011s
I20260709 13:03:41.901598  8657 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:03:41.901857  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.902169  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2712) took 1156 ms. Trace:
I20260709 13:03:41.895105  9322 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:41.913663 13925 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:41.914532 13876 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.915283 13848 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:41.915987 13848 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:41.917518  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 350210us
I20260709 13:03:41.645689 13937 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.922473  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.922706  8659 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:41.902577  8657 rpcz_store.cc:276] 0709 13:03:40.748733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.883213 (+134480us) service_pool.cc:224] Handling call
0709 13:03:41.902108 (+1018895us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:41.867352 13930 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:41.898033 13856 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.438s	user 0.020s	sys 0.044s
I20260709 13:03:41.948575  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2956) took 1778 ms. Trace:
W20260709 13:03:41.945080  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 324362us
W20260709 13:03:41.895957  8549 connection.cc:256] RPC call timeout handler was delayed by -1.101181978999989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
I20260709 13:03:41.933753  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.922843  8732 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:41.942549  8658 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:41.928938 13723 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.951692 13992 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:41.952046 13848 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:41.952682  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62090.4us
I20260709 13:03:41.646947 13756 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.928858  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:41.927574 13867 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.527s	user 0.012s	sys 0.000s
I20260709 13:03:41.966248 13328 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.966356 13904 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:41.966266  9327 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:41.968014 13904 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.968819  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2955) took 1812 ms. Trace:
I20260709 13:03:41.969266  8655 rpcz_store.cc:276] 0709 13:03:40.156159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.231101 (+ 74942us) service_pool.cc:224] Handling call
0709 13:03:40.231488 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:41.467520 (+1236032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.968546 (+501026us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000c80d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:41.968767 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2390514432}
I20260709 13:03:41.964587 13764 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.970463 13764 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:41.958415 13992 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:41.974138  8653 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:41.974321 13992 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:41.976644  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2942) took 1929 ms. Trace:
I20260709 13:03:41.961544 13912 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:41.978122 13912 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:41.956377  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.982939 13830 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:41.921293 13940 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.515s	user 0.005s	sys 0.000s
I20260709 13:03:41.989569 13640 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.970470  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.989114  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:41.991600  9329 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:41.991942  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:41.992535  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:41.993119  9329 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
W20260709 13:03:41.994060 13858 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.018s	user 0.012s	sys 0.001s
I20260709 13:03:41.994397  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:41.994992  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:41.995121  9329 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:41.924418 13824 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.952581  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.931593 13838 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.965793  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2947) took 1912 ms. Trace:
I20260709 13:03:41.966454 13867 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.996769  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.955659  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:41.947116 13908 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:41.998203  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 79114.7us
I20260709 13:03:41.998541 13867 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:41.998750 13712 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.633770 13956 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:41.995230  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:41.932845 13938 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:41.999480 13712 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:42.006341  9330 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:41.956614  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:41.999878  9323 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:42.007097  8549 connection.cc:256] RPC call timeout handler was delayed by -1.0276216719998956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
I20260709 13:03:41.999720  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:41.987306 13856 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.977006  8653 rpcz_store.cc:276] 0709 13:03:40.046847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.046999 (+   152us) service_pool.cc:224] Handling call
0709 13:03:40.047333 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:41.974660 (+1927327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.976366 (+  1706us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b5400380254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:41.976586 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4430188288}
I20260709 13:03:42.010749  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.002974  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.990177 13640 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:41.994495  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:41.954825 13930 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:41.953948  8650 rpcz_store.cc:276] 0709 13:03:40.170560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.264531 (+ 93971us) service_pool.cc:224] Handling call
0709 13:03:40.264918 (+   387us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:41.264867 (+999949us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:41.264924 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:41.265949 (+  1025us) log.cc:844] Serialized 52 byte log entry
0709 13:03:41.551128 (+285179us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:41.551310 (+   182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.558630 (+  7320us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.934370 (+375740us) raft_consensus.cc:1685] finished
0709 13:03:41.934388 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.948515 (+ 14127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25163,"raft.queue_time_us":112076,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":3033,"thread_start_us":1243873,"threads_started":2,"wal-append.queue_time_us":665112}
I20260709 13:03:42.009450  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:41.996340  8651 rpcz_store.cc:276] 0709 13:03:40.053021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.053161 (+   140us) service_pool.cc:224] Handling call
0709 13:03:40.053513 (+   352us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:40.055003 (+  1490us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:40.198737 (+143734us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b180044e6c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:03:41.335998 (+1137261us) log.cc:844] Serialized 14 byte log entry
0709 13:03:41.606382 (+270384us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:41.618443 (+ 12061us) log.cc:844] Serialized 52 byte log entry
0709 13:03:41.618609 (+   166us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:41.618654 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.618775 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.965483 (+346708us) raft_consensus.cc:1685] finished
0709 13:03:41.965580 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.965733 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15548,"prepare.queue_time_us":738926,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":16346,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":21449,"spinlock_wait_cycles":8084352,"thread_start_us":1293118,"threads_started":2,"wal-append.queue_time_us":600030}
I20260709 13:03:42.013345  9331 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:42.016124  8657 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:42.019377  9326 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 13 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:03:42.014889 13712 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:42.022961  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:42.025161  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:42.033239  9329 raft_consensus.cc:2368] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
I20260709 13:03:42.032924  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.040776  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.040550  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.043522  8652 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:42.039194  8698 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:42.048290 13991 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.046732  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.046517  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.052349  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.052645  8659 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:42.054352  8658 raft_consensus.cc:2368] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:03:41.994514 13858 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.059919 13938 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:42.033661 13876 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.038331 13856 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.068559  9331 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:42.062041  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.041527 13912 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.052981  8651 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:42.074534  8652 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:42.066934  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71057.6us
I20260709 13:03:42.019066 13830 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.029739 13751 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.076532  8653 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:42.048951 13991 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.096313  8658 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:03:42.104050  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 88723.6us
I20260709 13:03:42.057142 13756 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.109200  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.110423  8656 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:42.098541 13856 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.099664 13751 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:42.119113  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:42.092247 13876 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.124643  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:42.123133 13904 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:42.127810 13858 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.131914  9330 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:42.122453 13830 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.135073 13830 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:42.135768 13946 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:42.137642  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:42.141811 13751 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.144134 13751 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.146003 13756 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.117516 13640 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.150369  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:42.151422 13986 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 13:03:42.131824  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:42.162281 13938 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:42.166699 13938 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:42.167130 13938 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:42.169529 13956 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:42.167526  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:42.170439 13956 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:42.170921 13908 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:42.171553 13838 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:42.171471 13956 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.172231 13838 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:42.173142 13908 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:42.173126 13838 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.160508 13824 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:42.161221 13712 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:42.162973 13756 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:42.181291  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 559176us
I20260709 13:03:42.189894  8697 leader_election.cc:400] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
I20260709 13:03:42.190299  8697 leader_election.cc:403] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:42.194239 13991 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.194677  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.198431  8730 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:42.195168 13516 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:03:42.178954 13824 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:42.180429 13712 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:42.200158  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:42.181530 13908 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:42.203662  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53407.6us
I20260709 13:03:42.164425 13992 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.221361  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.222688  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:42.204128 13824 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:42.217439 13712 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:42.244851  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:42.249972  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.262208  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:42.264796  8731 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:42.301802  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57065.9us
I20260709 13:03:42.290606  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:42.310004  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57776.1us
I20260709 13:03:42.227572 13756 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.318197  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.191408 13920 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
W20260709 13:03:42.320405 13989 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
I20260709 13:03:42.348620  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.263608 13856 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.234709 13723 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:42.350220  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 104168us
I20260709 13:03:42.223572  9330 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea2142382a5d4439915e07c70837c688"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:42.363696  9330 raft_consensus.cc:606] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:42.356002  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 239504us
I20260709 13:03:42.364080  9330 raft_consensus.cc:614] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:42.361865  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:42.359373  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:42.363058  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.367527  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.369323  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.370213  8730 raft_consensus.cc:2410] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 13:03:42.368228  8728 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:42.371735  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:42.367947  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.376518 13743 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:42.378557  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.379392  8578 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:42.379521 13930 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.380378  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.380595  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.380896  8655 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:42.381141  8583 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:42.382005  8584 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:42.382203  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:42.382366  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.382902  8726 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:42.383092  8576 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:42.385591  8729 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:42.386184 13856 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.388105 13908 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.389731  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2700) took 1828 ms. Trace:
I20260709 13:03:42.390859 13723 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:42.390120  8654 rpcz_store.cc:276] 0709 13:03:40.561072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:41.384990 (+823918us) service_pool.cc:224] Handling call
0709 13:03:41.397802 (+ 12812us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:41.439194 (+ 41392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:41.439245 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:41.472382 (+ 33137us) log.cc:844] Serialized 26 byte log entry
0709 13:03:41.868975 (+396593us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:41.869040 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.869143 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:42.389504 (+520361us) raft_consensus.cc:1685] finished
0709 13:03:42.389524 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:42.389657 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53979,"raft.queue_time_us":29146,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":14280,"thread_start_us":329732,"threads_started":1,"wal-append.queue_time_us":547823}
I20260709 13:03:42.392023 13723 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.393792  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.381148  8579 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:42.386907  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.363131 13328 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:42.400223  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:42.405624  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:42.405637  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44787.8us
I20260709 13:03:42.408900  8575 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:42.412551  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.413287  8724 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:42.416775  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.417197  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.417539  8576 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:42.417938  8577 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:42.422180  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:42.408969  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:42.448935  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.399927 13743 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:42.452507  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 81657.6us
I20260709 13:03:42.396867 13996 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:42.427917  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:03:42.416406 13925 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.470642  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:42.471506  8724 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:42.455650  8659 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:42.459951 14002 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
W20260709 13:03:42.460968  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 220369us
I20260709 13:03:42.464690  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 861) took 1005 ms. Trace:
I20260709 13:03:42.472579  8655 rpcz_store.cc:276] 0709 13:03:41.459380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:41.991670 (+532290us) service_pool.cc:224] Handling call
0709 13:03:42.464478 (+472808us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:42.464627 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":892556032}
I20260709 13:03:42.465657 13743 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.466771  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:42.455312 13991 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:42.427825 13930 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:42.427239 13876 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:42.482577  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.473925  8729 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:42.489635  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:42.506562  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 52871.1us
I20260709 13:03:42.417025 13937 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 13:03:42.507697  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:42.499693  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:42.491437 13848 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:42.523435  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.537691  8655 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:03:42.532838  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.538141  9296 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:42.543884  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:42.545197  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.493731 13992 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:42.495316 13858 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:42.551899  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.552920  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.553889  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.451304 13739 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:03:42.546113  8652 raft_consensus.cc:2376] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14. Re-sending same reply.
I20260709 13:03:42.560171  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.561362  8651 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:42.524104 13328 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:42.496433 13920 consensus_queue.cc:1243] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:42.568694  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.569592  8650 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:42.563362  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.564327  8579 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:42.556955  8658 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:42.571045  8653 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:42.532371 13848 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:42.495247 13925 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:42.491955 13996 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.573861  8656 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
W20260709 13:03:42.862576 13712 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.569s	user 0.011s	sys 0.000s
W20260709 13:03:42.860545 13516 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.657s	user 0.009s	sys 0.000s
I20260709 13:03:42.923702 13516 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
I20260709 13:03:42.923062 13712 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:42.642724  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.925082  8727 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:42.926141  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.927490  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.928331  8657 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:42.850803 14003 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.208s	user 0.000s	sys 0.000s
I20260709 13:03:42.614847 14000 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:42.565819 13991 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:42.931397 13925 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:42.931841  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 382848us
I20260709 13:03:42.958115  8653 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:42.600320  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:42.932880  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 413162us
I20260709 13:03:42.552367 13772 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:42.578534  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71228.6us
I20260709 13:03:42.965448  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:42.924353 13996 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:42.816172  8726 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:03:42.929260  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.615533  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:42.923074  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.967109  8659 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:42.939806  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.944941 13848 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:42.540855 13930 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:42.671566 13995 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 13:03:42.947217 13897 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.939889  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:42.672204 13998 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260709 13:03:42.949862  8725 raft_consensus.cc:2410] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:03:42.961820 13991 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:42.963876  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 475177us
W20260709 13:03:42.973592 13824 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.683s	user 0.010s	sys 0.002s
I20260709 13:03:42.970748  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.971496  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.933972 13956 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.967130  8656 raft_consensus.cc:2376] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:03:42.662572 14014 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:43.006846 14014 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:43.007728 13756 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:43.007920 14014 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.009155 14014 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:42.969481  8657 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:42.974077  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.012918  8656 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:43.003160  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:43.014770  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:42.975247  8582 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:42.976363  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.976641  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:42.976825  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:42.977344  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.977734  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:42.978147  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.978310 13996 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:42.978533  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:43.019026  8656 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:42.979472  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.019917  8581 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:42.980183  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.980952  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.846879 13328 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:42.957837 13938 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.790s	user 0.009s	sys 0.005s
I20260709 13:03:43.021116  8584 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:42.969933  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:43.001639 13991 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:43.005671  8583 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:43.006296  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:43.017225  8579 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:43.017253  8655 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:43.005192  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.018591  8577 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:42.947288  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:43.026441  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 49695.8us
I20260709 13:03:43.026717  8576 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:43.028240 13938 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.031002 13938 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:43.033088  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 85890.2us
I20260709 13:03:43.034933  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:43.037221  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:43.037101 13772 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:43.042999  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:43.047459  8724 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:43.045730  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:43.049450  8728 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:43.051712  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:43.052587  9331 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:43.020103 13992 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:43.023931 13996 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.023623 13756 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:43.051577  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.003803 13930 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.042194 13993 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:43.010219 13824 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.082882  9331 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260709 13:03:43.083537  9331 pending_rounds.cc:85] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 10
I20260709 13:03:43.088865  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:43.014943 13956 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.089357  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:43.090262  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:42.999701 13897 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.065079  8575 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:43.063361 13996 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:43.083849  9331 pending_rounds.cc:107] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 13:03:43.143656  9331 raft_consensus.cc:2889] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:43.063189 13756 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:43.146903 13739 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:43.147468 13739 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:43.151017  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82424.6us
I20260709 13:03:43.151839  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:43.152513 13897 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:43.053256 13838 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.101s
I20260709 13:03:43.153155 13897 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:43.057780 14002 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:43.138731  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.184032  8575 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:43.139551  8728 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:43.141974  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:43.086488 13920 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:43.060910 13705 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:43.156579  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:43.216092  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.068030 13858 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.091s
I20260709 13:03:43.154165 13751 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.216858  8581 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
W20260709 13:03:43.216858  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 220991us
I20260709 13:03:43.088038 13993 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:43.148675 13739 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:43.184216  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 178012us
W20260709 13:03:43.187379  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 121285us
I20260709 13:03:43.057039 13938 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 13:03:43.231230  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:43.234634  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:43.222555  8655 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:43.182794 13897 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:43.236191 13897 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:43.237515 13751 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.238550 13938 consensus_queue.cc:1243] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:43.228153  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:43.229176  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.242244  8732 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:43.235167 14000 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:43.155700 13756 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:43.181958  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 120328us
I20260709 13:03:43.242524  8623 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:43.187275 13920 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.210273 13925 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:43.223582 13838 consensus_queue.cc:1243] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:43.259125  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:43.229105 13993 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:43.224474 13705 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.267154  8581 raft_consensus.cc:2376] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12. Re-sending same reply.
I20260709 13:03:43.258077 13856 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:43.322656  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.325740  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.462364  8575 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:43.257252 13920 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:43.260365 14021 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:43.458742  8732 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:03:43.461619  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 138464us
I20260709 13:03:43.459347  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:43.471794  8583 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:43.473510  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2528) took 1264 ms. Trace:
I20260709 13:03:43.483156  8732 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:43.484436  8725 rpcz_store.cc:276] 0709 13:03:42.197471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.199666 (+  2195us) service_pool.cc:224] Handling call
0709 13:03:43.327468 (+1127802us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b540008a2d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:43.461581 (+134113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1672682880}
I20260709 13:03:43.486790  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:43.463796 13705 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:43.469262 14029 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:43.501521 14029 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:43.472872 14002 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.502763  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.499706  8732 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.488251 13856 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:43.484105 14021 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:43.498293 14022 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.340s	user 0.009s	sys 0.000s
W20260709 13:03:43.506951  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.482544 14000 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:43.492400 13739 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:43.493909 13925 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.502543 13858 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:43.527632  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 285145us
W20260709 13:03:43.530438  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 299290us
I20260709 13:03:43.531077  8732 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:43.507069 13824 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:43.534572  8729 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:43.540385  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:43.507453 13956 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:43.536396  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:43.538435 13996 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:43.539871  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 80616.1us
I20260709 13:03:43.502048 14029 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:43.529448 13751 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:43.544242 13705 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.546389  8724 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:43.546770 13705 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:43.548269 14019 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.280s	user 0.000s	sys 0.008s
I20260709 13:03:43.550663 14002 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.550493 14021 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.552296  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 2971) took 1104 ms. Trace:
I20260709 13:03:43.562534  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2539) took 1210 ms. Trace:
I20260709 13:03:43.566221  8730 rpcz_store.cc:276] 0709 13:03:42.352424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.465489 (+113065us) service_pool.cc:224] Handling call
0709 13:03:42.465944 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:42.834093 (+368149us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:42.948232 (+114139us) log.cc:844] Serialized 14 byte log entry
0709 13:03:43.230239 (+282007us) log.cc:844] Serialized 14 byte log entry
0709 13:03:43.254384 (+ 24145us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:43.261891 (+  7507us) log.cc:844] Serialized 26 byte log entry
0709 13:03:43.468875 (+206984us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:43.468954 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:43.469068 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:43.562227 (+ 93159us) raft_consensus.cc:1685] finished
0709 13:03:43.562258 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:43.562446 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113566,"prepare.queue_time_us":116257,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":240252,"raft.run_cpu_time_us":548,"raft.run_wall_time_us":58072,"spinlock_wait_cycles":4546944,"thread_start_us":577248,"threads_started":2,"wal-append.queue_time_us":201210}
I20260709 13:03:43.552999  8654 rpcz_store.cc:276] 0709 13:03:42.447707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.447842 (+   135us) service_pool.cc:224] Handling call
0709 13:03:42.460074 (+ 12232us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:42.482270 (+ 22196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:42.482308 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:42.518970 (+ 36662us) log.cc:844] Serialized 52 byte log entry
0709 13:03:42.962818 (+443848us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:43.227236 (+264418us) log.cc:844] Serialized 14 byte log entry
0709 13:03:43.227886 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:43.227969 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:43.551926 (+323957us) raft_consensus.cc:1685] finished
0709 13:03:43.551961 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:43.552204 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":374598,"prepare.queue_time_us":529340,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":197721,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":32600,"thread_start_us":626739,"threads_started":2,"wal-append.queue_time_us":514046}
I20260709 13:03:43.571235  8580 raft_consensus.cc:2368] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12.
I20260709 13:03:43.572031  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:43.572741  8732 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 13:03:43.574402  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.574244  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:43.575445  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71327.6us
I20260709 13:03:43.576673  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:43.597792  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 990) took 1368 ms. Trace:
I20260709 13:03:43.593945  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:43.600173  8732 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:43.598184  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:43.601153  8730 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:43.602137  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:43.602308  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 418950us
I20260709 13:03:43.603962  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:43.604686  8732 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:03:43.599274  8727 rpcz_store.cc:276] 0709 13:03:42.229332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.234002 (+  4670us) service_pool.cc:224] Handling call
0709 13:03:43.597563 (+1363561us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54000e9f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:43.597726 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1550498176}
I20260709 13:03:43.608582  8724 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:43.614068  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:43.553385 14002 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:43.612170  8696 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.517s (SENT)
I20260709 13:03:43.569234 14018 raft_consensus.cc:2764] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
W20260709 13:03:43.698460  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 86351.3us
I20260709 13:03:43.563380 13996 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:43.552337 13858 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:43.577873 14000 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:03:43.627935 14014 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.610s	user 0.006s	sys 0.006s
W20260709 13:03:43.707768  8696 leader_election.cc:336] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.578s (SENT)
W20260709 13:03:43.709715  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 106302us
W20260709 13:03:43.715050  8696 leader_election.cc:336] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.588s (SENT)
I20260709 13:03:43.715847  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:43.718007  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:43.534634 13925 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.721566 13925 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:43.711124  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 223622us
I20260709 13:03:43.542670 13723 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.162s
I20260709 13:03:43.620682 13705 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:43.621142 14038 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:43.749650  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 205774us
I20260709 13:03:43.581346 13993 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.699230 14036 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:43.754547  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.747184  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:43.751783  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 215467us
I20260709 13:03:43.751176  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.743602  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:43.761209  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.710965 14000 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:43.755503  8575 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 7.
I20260709 13:03:43.743423 14002 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:43.772248  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:43.772869  8581 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
W20260709 13:03:43.776731  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.725962 13897 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.771242 14027 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:43.771556 14000 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:43.802453 13772 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:43.803018 13772 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:43.803946 13772 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:43.792091  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 82193.6us
I20260709 13:03:43.778223 13340 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:43.808549  8698 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:03:43.795199  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44751.4us
W20260709 13:03:43.819105  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 104098us
W20260709 13:03:43.819793  8696 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.623s (SENT)
I20260709 13:03:43.774977 14014 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
W20260709 13:03:43.822036  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104100us
I20260709 13:03:43.822811  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:43.809914  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.745616 14018 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 13:03:43.806236 13772 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:43.776644 13756 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.827430  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:43.827888 14027 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:43.725162 14040 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.101s
I20260709 13:03:43.850062  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2534) took 1498 ms. Trace:
W20260709 13:03:43.847683  8696 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.671s (SENT)
I20260709 13:03:43.850808  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:43.830962 13340 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:43.849151  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:43.850601  8731 rpcz_store.cc:276] 0709 13:03:42.351273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.402018 (+ 50745us) service_pool.cc:224] Handling call
0709 13:03:42.419266 (+ 17248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:42.419503 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:42.459503 (+ 40000us) log.cc:844] Serialized 14 byte log entry
0709 13:03:43.323686 (+864183us) spinlock_profiling.cc:243] Waited 9.98 ms on lock 0x7b1800444460. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:03:43.839518 (+515832us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:43.839612 (+    94us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:43.849285 (+  9673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:43.849424 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:43.849611 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99368,"prepare.queue_time_us":38261,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":58352,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":27953,"spinlock_wait_cycles":22944000,"thread_start_us":515458,"threads_started":2,"wal-append.queue_time_us":1253761}
I20260709 13:03:43.857098 13993 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:43.827921 13723 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 13:03:43.852049 14027 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:43.871083  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:43.869747  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 74698.8us
I20260709 13:03:43.808427 13858 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.870584  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:43.867444 13824 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:43.880548 14059 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:43.881193 14059 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:43.881723 14059 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:43.881561 14046 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:03:43.840322 14052 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:03:43.885829 13991 consensus_meta.cc:287] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.839s	user 0.009s	sys 0.003s
I20260709 13:03:43.885973 14052 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:43.886566 14052 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:43.886426 13991 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:43.879284 14047 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:43.890430  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:43.890681  8730 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:43.891551  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:43.894120  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.896744  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:43.897763  8655 raft_consensus.cc:2368] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
I20260709 13:03:43.896884 13756 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:43.904313  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55277.3us
I20260709 13:03:43.864343 14036 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:43.907073  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:43.903426  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:43.914921  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2795) took 1289 ms (client timeout 1607 ms). Trace:
W20260709 13:03:43.915429  8733 rpcz_store.cc:269] 0709 13:03:42.635720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.922295 (+286575us) service_pool.cc:224] Handling call
0709 13:03:43.906902 (+984607us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:03:43.914842 (+  7940us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6242560}
W20260709 13:03:43.919406  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 90749us
I20260709 13:03:43.921769  9331 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:43.939917  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.940973  9324 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:03:43.941852  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.941957  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:43.939977  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.943338  9329 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:43.945667  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:43.946056  8728 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:43.946499  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:43.947773  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.949504  8655 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:43.948501  9326 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
W20260709 13:03:43.950145  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 72316us
I20260709 13:03:43.950845  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:43.949909  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.921633 14057 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:43.952107  9328 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:43.952728  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 60780.5us
I20260709 13:03:43.951478  8654 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:43.947932 14054 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:43.962927 13991 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:43.963135  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.960032 14057 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.963354  9328 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:43.969483 14057 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:43.964039  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 148217us
I20260709 13:03:43.947891 13340 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.970460 13340 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.964972  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:43.971320  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:43.971657  8730 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:43.963815  9326 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:43.972330  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:43.972970  9331 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
W20260709 13:03:43.980252 14012 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 13:03:43.986490 14029 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:43.989423 14027 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:44.013983  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.014196  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:43.974777 14014 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:44.012578 13848 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.015s	user 0.010s	sys 0.003s
I20260709 13:03:44.034488  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:44.039327  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 68087.5us
I20260709 13:03:44.011008 13897 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:44.046603  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:44.056892  9323 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:44.054243  8730 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:44.058826  8658 raft_consensus.cc:1217] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Deduplicated request from leader. Original: 6.4->[10.5-10.5]   Dedup: 10.5->[]
I20260709 13:03:44.061738  8658 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 13:03:44.043339 14002 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.064404  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.019979 14029 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:44.064256 14061 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 13:03:44.068092  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:44.070405  9323 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:03:44.074311  9330 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:44.076548  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 92927.2us
I20260709 13:03:44.078372  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:44.031374 14000 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.008754 14036 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.031548 14052 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.084465  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:44.084898  9330 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.058404 13848 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.060947 14014 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.094439  9327 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:44.101891  9322 raft_consensus.cc:2368] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
I20260709 13:03:44.103048 13848 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:44.106245  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.083040 13948 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:44.087069 13991 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.094028  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.086407 13340 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:44.116465  8696 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.768s (SENT)
I20260709 13:03:44.117085  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:44.117308  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:44.122296  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:44.126279 14029 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:44.082223 14062 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:03:44.126766 14029 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.127203 14029 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:44.127835 14000 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:44.129801  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:44.118111 13920 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.618s	user 0.008s	sys 0.003s
W20260709 13:03:44.118333 13856 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.599s	user 0.011s	sys 0.001s
I20260709 13:03:44.132876  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.141634  9330 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:44.142598  9323 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:44.144964  8698 leader_election.cc:400] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11.
I20260709 13:03:44.153103  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:44.115185 13948 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.159849  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:44.170192 14051 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:44.171636  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.176537  8657 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:44.177681  9323 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:44.143886 13340 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:44.182384  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49557.8us
W20260709 13:03:44.182391  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65976.1us
I20260709 13:03:44.188584  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.151656 14050 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:44.167395 13856 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:44.167650 14000 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.188226  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:44.161170 13756 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:44.185230  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:44.189777  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.174358 14049 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:44.197237  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:44.197824  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 91679.2us
I20260709 13:03:44.187355 13739 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 13:03:44.191419 14044 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.019s
I20260709 13:03:44.203233 14044 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:44.203626 14044 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.156038 13920 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.206321 13920 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:44.207146  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.185911 13772 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:44.208423  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.208730 13772 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:44.209794 13772 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:44.211071 14064 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.165s	user 0.001s	sys 0.001s
I20260709 13:03:44.212154 14000 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.214289 13992 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.214517 14000 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:44.214847 13992 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.217751 14055 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:44.218129  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.219213  8659 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.182138 14057 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:44.225199  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45410.5us
I20260709 13:03:44.227890  8657 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:03:44.229796  8650 raft_consensus.cc:2368] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:44.199656  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.246694  8655 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
W20260709 13:03:44.247385  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60572.7us
I20260709 13:03:44.218333 14055 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:44.249342  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.250723  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:44.252925  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.198468 14049 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:44.253906  8654 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:44.225277 13723 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:44.258998  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:44.260564 14014 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:44.264580 14041 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:44.265431 14063 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 13:03:44.246438 13692 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:44.268169 14044 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.270099  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.271579 13824 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:44.274108 14054 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:03:44.276333 14065 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 13:03:44.251544 14055 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.276530 14052 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:44.265715 13948 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.251763 13848 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:44.254949 13858 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:44.286130  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:44.257859 14049 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:44.279806 14050 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:44.278329 13723 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:44.301065  8659 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:44.249279 13756 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.284301 13772 raft_consensus.cc:2764] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
I20260709 13:03:44.309552  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:44.309824 14044 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.293200 14054 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:44.284672 13692 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:44.290159 14041 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:44.313984  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55017.2us
I20260709 13:03:44.316681  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 990) took 1342 ms. Trace:
W20260709 13:03:44.317819  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69962.7us
I20260709 13:03:44.320804  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:44.305697 13848 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:44.338373  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:44.245827 14059 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.340976  8580 rpcz_store.cc:276] 0709 13:03:42.971670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.976941 (+  5271us) service_pool.cc:224] Handling call
0709 13:03:44.312177 (+1335236us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b5400343fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:44.313853 (+  1676us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262258560}
I20260709 13:03:44.322904 13723 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:44.345499  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 122322us
I20260709 13:03:44.343752  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:44.348827 13692 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:44.382278  8696 leader_election.cc:336] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.930s (SENT)
I20260709 13:03:44.349419 14021 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.385622 13692 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:44.385740  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43773.2us
I20260709 13:03:44.386636  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.387853  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.388895 14052 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:44.389309 14052 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:44.389747 14052 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.391247 13858 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.394078  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.352299 13992 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.339591 14041 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.356890 14000 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:44.404927  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2793) took 1855 ms. Trace:
I20260709 13:03:44.405489  8728 rpcz_store.cc:276] 0709 13:03:42.549127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.207313 (+658186us) service_pool.cc:224] Handling call
0709 13:03:43.210509 (+  3196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:43.946414 (+735905us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:44.404660 (+458246us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54004722d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:44.404857 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1686876544}
I20260709 13:03:44.358120 14002 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:44.357841 14048 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:44.407742 13897 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:44.407915 14048 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:44.408244 13897 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:44.408576 14048 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.361531 13772 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:44.358605 13948 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.367718 13756 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:44.368136  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2789) took 1966 ms. Trace:
I20260709 13:03:44.368638  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:44.333791 14054 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.213338  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:44.373528  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:44.332641 14014 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
W20260709 13:03:44.376097  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.344116 13848 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:44.430158  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.430378  8729 rpcz_store.cc:276] 0709 13:03:42.401656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.495457 (+ 93801us) service_pool.cc:224] Handling call
0709 13:03:42.532071 (+ 36614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:43.541851 (+1009780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:44.110470 (+568619us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b54000db220. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c19a02 00007f1409c15dd6 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:44.367904 (+257434us) spinlock_profiling.cc:243] Waited 1e+03 ms on lock 0x7b5400417154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:44.368081 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2302515968}
I20260709 13:03:44.408739 14050 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:44.435629 14050 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:44.436187 14050 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.409328 13897 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.440291 13897 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.437139 14050 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.325105 14034 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 13:03:44.419044 13992 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:44.443456 14034 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:44.416420  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:44.432687  8698 leader_election.cc:400] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:44.414021 14055 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:44.433320  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63910.3us
I20260709 13:03:44.412936 14049 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.445242 14016 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:44.451406 14016 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:44.452872 13712 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:44.431164 13993 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:44.453999  8730 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:44.444090 14034 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.454659 14055 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:44.431067 14047 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.455442 14034 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.456897 14034 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:44.409462 13772 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.460981 13772 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:44.463233 14029 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.463541 14005 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:44.463017 14024 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Committing config change with OpId 9.9: config changed from index 7 to 9, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:44.465700 14029 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:44.465266 14028 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:03:44.469720 14070 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.204s	user 0.002s	sys 0.002s
I20260709 13:03:44.470654  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:44.470999  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:44.472795  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:44.473588 13756 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 13:03:44.475124 14073 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.219s	user 0.002s	sys 0.000s
I20260709 13:03:44.449766  9324 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:44.475685 13756 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:44.476270 13756 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.448243  8698 leader_election.cc:403] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:44.452410 14016 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.475056 14000 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:44.473923  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 1234) took 1439 ms. Trace:
I20260709 13:03:44.478482 14000 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.452901 14049 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:44.478960  9322 rpcz_store.cc:276] 0709 13:03:43.034182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.034672 (+   490us) service_pool.cc:224] Handling call
0709 13:03:44.473691 (+1439019us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54007ab554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:44.473835 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2451908480}
I20260709 13:03:44.479671  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:44.480224 13751 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:44.480820 13751 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.479769 14000 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.481607 14044 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:44.481714  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:44.482180 14044 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.481881 13751 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.482640 14044 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.483031 14055 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.481189  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:44.478521  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48452.2us
I20260709 13:03:44.487038  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.487228 14002 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:44.487793 14000 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.491849  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.492560  8658 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.494243  8582 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 13:03:44.494655  8582 pending_rounds.cc:85] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 8
I20260709 13:03:44.494908  8582 pending_rounds.cc:107] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 13:03:44.494809 13858 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:44.495391  8582 raft_consensus.cc:2889] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:44.497190  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.498070  8582 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:44.498152 14016 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.477242 13756 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:44.478899  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 94195.9us
I20260709 13:03:44.501776  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:44.503036  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:44.346941  8548 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:44.505337  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 75252.5us
W20260709 13:03:44.508085  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.422780 13991 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:44.487929 14002 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.517904 13930 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:44.517728 14002 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.518612  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:44.522119  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 175252us
I20260709 13:03:44.522905  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:44.517479 14044 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.523708  8658 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:44.525826  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.524585  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:44.526638  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.529801 13992 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:44.531983  8622 leader_election.cc:336] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.542s (SENT)
I20260709 13:03:44.532538  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.535223  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:44.551326  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:44.552198  8731 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:44.557677  8622 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.567s (SENT)
I20260709 13:03:44.557936  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:44.558259  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.561869  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:44.448664 14057 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:44.527596  8659 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 7 to 9, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:44.571846  8733 raft_consensus.cc:1217] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Deduplicated request from leader. Original: 7.8->[10.9-13.10]   Dedup: 10.9->[13.10-13.10]
I20260709 13:03:44.572696  8733 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: [10.9-13.10]
W20260709 13:03:44.574043  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 49524.2us
I20260709 13:03:44.577131  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.578879  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:44.520710 14032 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:03:44.594368  8724 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
W20260709 13:03:44.594587  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 212365us
W20260709 13:03:44.606508  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:44.607050  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:44.607666  8696 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.959s (SENT)
I20260709 13:03:44.607833  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.580245  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:44.616308  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:44.616789  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:44.617647  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 94818.7us
W20260709 13:03:44.618577  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60953.4us
I20260709 13:03:44.618705  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:44.618844  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:44.619648  8733 raft_consensus.cc:2410] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:03:44.620417  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.556970 13993 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:44.627454  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:44.554214 14040 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:44.621672  8727 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:03:44.533803 13824 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.617110  9329 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:44.562304 13992 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.568864 13937 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:44.564780 13930 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:03:44.637549  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 111802us
I20260709 13:03:44.651228  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.487222 14052 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.645026  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.549480 13772 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.088s
I20260709 13:03:44.612923 13328 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.655959  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:44.657912  8729 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:44.619814 13858 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:44.619695 14036 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:44.665699  9294 leader_election.cc:336] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.811s (SENT)
I20260709 13:03:44.667232  8579 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:44.669948 14050 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:44.643268 14040 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.643710 14032 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:44.674067  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.642153 14028 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:44.608462 13938 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 13:03:44.679364  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.653187 13992 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:44.608819 14027 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 13:03:44.410192 14048 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.667363 13858 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:44.673172 14057 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:44.720652  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 61650.2us
W20260709 13:03:44.719851  8622 leader_election.cc:336] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.731s (SENT)
I20260709 13:03:44.743588  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.684417 14032 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:44.730000 14077 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:03:44.685739 14040 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:44.731513  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65807.5us
I20260709 13:03:44.701222 13751 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:44.670399 14050 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.728185 13937 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:44.766670  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.766919 13937 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.767390  8584 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:44.681981 14046 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:44.769641 14046 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:44.770356 14046 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:44.773923  8622 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.534s (SENT)
I20260709 13:03:44.774538  8622 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:44.650537 13824 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.777971 13824 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:44.778829 13751 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:44.779333 13751 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.782939 14050 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.784651 14050 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:44.681800 14036 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:44.756417  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 129046us
W20260709 13:03:44.790959  8696 leader_election.cc:336] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.821s (SENT)
I20260709 13:03:44.791483  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.794994  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:44.796054  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:44.797040  9327 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:44.668277 13328 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.745635 14057 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:44.746104 13858 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:44.774175 14086 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 13:03:44.738375  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.671916 13848 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.744323 13993 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:44.799966  8622 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.795s (SENT)
I20260709 13:03:44.817587  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:44.765410  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.802016  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:44.819700  8655 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:44.820215  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.820896  8657 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:44.797277 13692 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:44.807600  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.823510  8584 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:44.763517 14018 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:44.824760  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.825580  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:44.827521  8659 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:03:44.828047 14089 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:44.828691 14089 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:44.830245  8657 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:44.832983 14029 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:44.834512 14030 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:44.834933 14030 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:44.835860 14030 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:44.812117 14057 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:44.840235  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:44.802058  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:44.753060 14032 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:44.815763  8579 raft_consensus.cc:2368] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:03:44.756577 14044 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:44.813814 13824 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:44.807425 14036 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.841518 14052 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:44.842993  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.843453  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.848238  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.849071  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.849257  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:44.769706 14075 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:03:44.874619  8584 raft_consensus.cc:2368] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12.
I20260709 13:03:44.848075 13692 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:44.850770 14029 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:44.851263  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.851863  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.852708  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.876247  8577 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:03:44.853363 13743 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:44.860188  8580 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:44.862762  8731 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:03:44.874128  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.880280  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.888106  8584 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:44.868723  8478 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 12, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:44.882992  8581 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:44.883621  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.889993  8583 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:44.892405  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.849632 14059 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:44.884754 13743 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.902786  8729 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:03:44.903118  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:44.902904  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:44.904452  8622 leader_election.cc:336] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.915s (SENT)
I20260709 13:03:44.905856  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:44.906802  8577 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:44.907909  8548 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:44.906843  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:44.910740  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:44.885061 14048 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:44.912580 14005 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:44.888541 14052 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:44.886217 14047 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:44.947775  8622 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.947s (SENT)
I20260709 13:03:44.959069  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:44.829730 14089 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:44.960705  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:44.968346  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65290.4us
W20260709 13:03:44.917752 14021 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.520s	user 0.008s	sys 0.002s
I20260709 13:03:44.910243 13743 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:44.987565  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:45.058174  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.010823  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:44.912626 14044 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:44.888756 13993 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 13:03:44.912925 13692 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:45.082901  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:45.087374  8575 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:45.061905  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 416920us
I20260709 13:03:45.070358  8726 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:44.916306 14093 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:44.959046 13848 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:44.884354 13723 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:45.095610  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.097143  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:45.097903  8575 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:45.099283  9326 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:45.070715 14085 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 13:03:45.103116  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:45.107864  8584 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:44.972661 14005 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:45.057945 13772 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:45.090975  9329 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 7 to 9, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:45.118140  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:45.011066 14052 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:45.127048  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:45.096380 14095 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.139014  8584 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 13:03:45.100103 13692 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:45.129529  8575 raft_consensus.cc:2368] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:03:45.133153  9322 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:45.162081  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.166087  9325 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
W20260709 13:03:45.170102 13948 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.723s	user 0.002s	sys 0.008s
I20260709 13:03:45.172439 13723 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:45.174016 14005 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:45.179527 14054 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.728s	user 0.009s	sys 0.000s
I20260709 13:03:45.180076 14054 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.191079  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:45.196448  8584 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:45.200973  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:45.227049  8657 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:45.221663  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:45.235278  9322 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
W20260709 13:03:45.244560  8622 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.234s (SENT)
W20260709 13:03:45.256250  8725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.004s	sys 0.021s
I20260709 13:03:45.267833  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:45.267838  8725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.004s	sys 0.021s
W20260709 13:03:45.362429  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 463200us
I20260709 13:03:45.241343 13328 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:45.248179 14014 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.003s	sys 0.023s
I20260709 13:03:45.253813 14029 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:45.434719  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.146848 14095 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.354163 14036 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
I20260709 13:03:45.407395 14062 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.398501 13930 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:45.505155  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:45.508621 14027 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.510160 14109 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:45.666250  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.013s	sys 0.021s
W20260709 13:03:45.517630  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:45.509088  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.551456 14028 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.249714 14052 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.533627  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:45.534789  9322 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:45.233546 14002 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.009s	sys 0.019s
W20260709 13:03:45.546922  8696 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.881s (SENT)
I20260709 13:03:45.681154  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:45.551625  8581 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.009s	sys 0.015s
W20260709 13:03:45.681839  8581 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.010s	sys 0.015s
W20260709 13:03:45.573868  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 329332us
W20260709 13:03:45.659041  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:45.559003 13937 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.010s	sys 0.014s
W20260709 13:03:45.663981  8651 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.014s	sys 0.004s
I20260709 13:03:45.512868 14021 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.407004 13991 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:45.667332  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.013s	sys 0.021s
I20260709 13:03:45.670251  9325 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
W20260709 13:03:45.685223  8651 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.015s	sys 0.004s
I20260709 13:03:45.660964 14041 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:45.708307  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 161419us
W20260709 13:03:45.684356 14002 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.163s	user 0.009s	sys 0.020s
W20260709 13:03:45.724582  9327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.011s	sys 0.012s
W20260709 13:03:45.725063  9327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.011s	sys 0.013s
W20260709 13:03:45.691318 13937 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.011s	sys 0.015s
W20260709 13:03:45.698869  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.716576  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:45.696573  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 151636us
I20260709 13:03:45.729012  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:45.733539  8576 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:45.729004  8733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.006s	user 0.019s	sys 0.019s
I20260709 13:03:45.416517 13692 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:45.734059  8733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.011s	user 0.019s	sys 0.019s
I20260709 13:03:45.585690 14029 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.683843 13723 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:45.741730  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.000s	sys 0.040s
W20260709 13:03:45.742076 13756 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.011s	sys 0.015s
W20260709 13:03:45.758370  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.009s	sys 0.033s
W20260709 13:03:45.900007  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.017s	sys 0.041s
W20260709 13:03:45.758021  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.001s	sys 0.041s
W20260709 13:03:45.904014  9329 consensus_meta.cc:287] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.787s	user 0.018s	sys 0.001s
W20260709 13:03:45.904147  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 878154us
I20260709 13:03:45.741360  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2560) took 1112 ms. Trace:
I20260709 13:03:45.662519 14062 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:45.758576  8731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.032s
W20260709 13:03:45.913642  8731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.002s	sys 0.034s
I20260709 13:03:45.914616  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2565) took 1028 ms. Trace:
W20260709 13:03:45.852443  8583 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.952s	user 0.016s	sys 0.001s
W20260709 13:03:45.802634  9328 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.013s	sys 0.026s
I20260709 13:03:45.907323  8733 rpcz_store.cc:276] 0709 13:03:44.628627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.644528 (+ 15901us) service_pool.cc:224] Handling call
0709 13:03:44.662389 (+ 17861us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:44.662589 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:44.722490 (+ 59901us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.740673 (+1018183us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:45.741053 (+   380us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:45.741110 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.741178 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.741298 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":9394,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":10528,"raft.queue_time_us":105,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":16234,"thread_start_us":1076672,"threads_started":2,"wal-append.queue_time_us":1160678}
I20260709 13:03:45.915045  8731 rpcz_store.cc:276] 0709 13:03:44.886660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.126402 (+239742us) service_pool.cc:224] Handling call
0709 13:03:45.150398 (+ 23996us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.155994 (+  5596us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.156046 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.187588 (+ 31542us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.914208 (+726620us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:45.914257 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.914337 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:45.914384 (+    47us) raft_consensus.cc:1685] finished
0709 13:03:45.914401 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.914554 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5115,"raft.queue_time_us":22622,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":8159,"thread_start_us":686723,"threads_started":1,"wal-append.queue_time_us":680140}
W20260709 13:03:45.847611  8622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.529s (SENT) [suppressed 78 similar messages]
I20260709 13:03:45.916626  8583 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:45.714929 14116 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:45.885424 14095 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:45.895006  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 966) took 1313 ms. Trace:
I20260709 13:03:45.742477 14054 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:45.759687  8584 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 15 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:45.718247 14021 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.758133  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:45.684569 14028 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.679409 14109 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:45.916226  9328 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.013s	sys 0.026s
I20260709 13:03:45.939165  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 967) took 1357 ms. Trace:
I20260709 13:03:45.678891 14027 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:45.880187 14111 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.523s	user 0.002s	sys 0.000s
I20260709 13:03:45.938691  8576 raft_consensus.cc:2376] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:03:45.917273  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1012) took 1053 ms. Trace:
I20260709 13:03:45.943975  8583 rpcz_store.cc:276] 0709 13:03:44.863973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.883324 (+ 19351us) service_pool.cc:224] Handling call
0709 13:03:45.917210 (+1033886us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:45.946094  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:45.831151 13756 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.011s	sys 0.015s
I20260709 13:03:45.729735 13991 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:45.919626  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:45.927462  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:45.758754 13723 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.939612  9328 rpcz_store.cc:276] 0709 13:03:44.581805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.581926 (+   121us) service_pool.cc:224] Handling call
0709 13:03:44.582224 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:44.582491 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:44.582879 (+   388us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.074925 (+492046us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.938813 (+863888us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:45.938866 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:45.938910 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.938971 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.939108 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":151770,"prepare.queue_time_us":1795,"prepare.run_cpu_time_us":6574,"prepare.run_wall_time_us":13226,"raft.queue_time_us":131638,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":148,"thread_start_us":1170582,"threads_started":2,"wal-append.queue_time_us":1130237}
W20260709 13:03:45.967032 14040 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.168s	user 0.012s	sys 0.000s
I20260709 13:03:46.029137 14040 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.933320  9323 rpcz_store.cc:276] 0709 13:03:44.581387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.581555 (+   168us) service_pool.cc:224] Handling call
0709 13:03:44.617709 (+ 36154us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.617937 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.617967 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.637685 (+ 19718us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.110954 (+473269us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:45.111007 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.111090 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:45.894656 (+783566us) raft_consensus.cc:1685] finished
0709 13:03:45.894685 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.894921 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":435163,"threads_started":1,"wal-append.queue_time_us":927593}
I20260709 13:03:45.872226 14035 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:45.736377 14041 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:45.900650 14002 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:45.913537 14100 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.498s	user 0.000s	sys 0.001s
I20260709 13:03:45.948405  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:46.580524 14115 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.673s	user 0.011s	sys 0.007s
W20260709 13:03:46.010248 14092 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:03:46.011279  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:46.530776  8724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.016s	user 0.008s	sys 0.012s
W20260709 13:03:46.637174  8724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.122s	user 0.009s	sys 0.013s
I20260709 13:03:46.637413  8583 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:46.492424  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.010s	sys 0.007s
W20260709 13:03:46.637957  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.010s	sys 0.010s
W20260709 13:03:46.142083  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 399211us
I20260709 13:03:46.577373  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1011) took 1713 ms. Trace:
I20260709 13:03:46.639526  8581 rpcz_store.cc:276] 0709 13:03:44.863628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.893600 (+ 29972us) service_pool.cc:224] Handling call
0709 13:03:44.893927 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:44.894151 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:45.682196 (+788045us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.181103 (+498907us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:46.577063 (+395960us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:46.577119 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.577187 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.577312 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":990715,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":1278987,"threads_started":2}
I20260709 13:03:46.641659  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:46.576499 13723 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:46.257570 14024 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.036s	sys 0.057s
W20260709 13:03:46.582630  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:46.583110 14032 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.698s	user 0.011s	sys 0.034s
I20260709 13:03:46.650573 14032 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.653072 14032 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.571883  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 887) took 1752 ms. Trace:
I20260709 13:03:46.585786  9294 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:46.518958  9329 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.001s	sys 0.012s
W20260709 13:03:46.672631  9329 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.002s	sys 0.013s
W20260709 13:03:46.203817  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:46.492270 14089 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.000s	sys 0.016s
W20260709 13:03:45.947926 14034 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.491s	user 0.012s	sys 0.014s
W20260709 13:03:46.580117 14016 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 2.081s	user 0.016s	sys 0.014s
I20260709 13:03:46.579372 14021 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.569718 14040 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:46.577939 14050 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.793s	user 0.020s	sys 0.010s
W20260709 13:03:46.537871  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.002s	sys 0.003s
W20260709 13:03:46.573534 14102 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.666s	user 0.006s	sys 0.006s
W20260709 13:03:46.702275  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.002s	sys 0.003s
I20260709 13:03:46.702468  8575 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:46.569360 14057 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.731s	user 0.010s	sys 0.012s
W20260709 13:03:46.703181  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.703405 14057 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.704068  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 969) took 1614 ms. Trace:
I20260709 13:03:46.705348 14050 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.705735  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 968) took 2123 ms. Trace:
I20260709 13:03:46.704571  9329 rpcz_store.cc:276] 0709 13:03:45.089877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.090104 (+   227us) service_pool.cc:224] Handling call
0709 13:03:45.090515 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.090843 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:45.904459 (+813616us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.703382 (+798923us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.703618 (+   236us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:46.703682 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:46.703741 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.703830 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.703996 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31480,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":44,"raft.queue_time_us":5519,"thread_start_us":768331,"threads_started":1}
I20260709 13:03:46.705956 14050 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:46.706173  9330 rpcz_store.cc:276] 0709 13:03:44.582098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.582237 (+   139us) service_pool.cc:224] Handling call
0709 13:03:44.582629 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.612148 (+ 29519us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.612200 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.658209 (+ 46009us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.102424 (+444215us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:45.102489 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.102843 (+   354us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.705492 (+1602649us) raft_consensus.cc:1685] finished
0709 13:03:46.705518 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.705673 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29015,"raft.queue_time_us":29352,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":22413,"thread_start_us":379523,"threads_started":1,"wal-append.queue_time_us":1037647}
I20260709 13:03:46.708269  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:46.708559 14050 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:46.709112  9330 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 9: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:46.572849 13897 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 2.132s	user 0.013s	sys 0.010s
I20260709 13:03:46.568373 13756 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.710129  9324 raft_consensus.cc:2410] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 13:03:46.710041 13756 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:46.711263 14097 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.303s	user 0.003s	sys 0.000s
I20260709 13:03:46.606751  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:46.712731  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2545) took 2255 ms. Trace:
I20260709 13:03:46.713361  9328 raft_consensus.cc:2410] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:03:46.713147  8728 rpcz_store.cc:276] 0709 13:03:44.457355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.513546 (+ 56191us) service_pool.cc:224] Handling call
0709 13:03:44.555971 (+ 42425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.556265 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.556305 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.556904 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 13:03:44.959477 (+402573us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:44.959514 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:44.959582 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.659484 (+1699902us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:46.712387 (+ 52903us) raft_consensus.cc:1685] finished
0709 13:03:46.712421 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.712650 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17955712,"thread_start_us":313670,"threads_started":1,"wal-append.queue_time_us":586353}
I20260709 13:03:46.714598  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:46.716100  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:46.712755 13756 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:46.717003  9328 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260709 13:03:46.580482 14048 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.606s	user 0.014s	sys 0.017s
I20260709 13:03:46.717968  8578 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 9.
W20260709 13:03:46.598349  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.586179  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:46.719204  8659 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:46.614614  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1246) took 1518 ms. Trace:
W20260709 13:03:46.609297  9297 connection.cc:256] RPC call timeout handler was delayed by -1.3034843310000497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
W20260709 13:03:46.587723  8622 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.002s	sys 0.018s
W20260709 13:03:46.720641  8578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2894) took 2068 ms (client timeout 1588 ms). Trace:
I20260709 13:03:46.721199 13897 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:46.721532  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:46.721829 13897 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.722370 13991 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:46.571851 14055 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 2.088s	user 0.007s	sys 0.022s
I20260709 13:03:46.724133 13897 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:46.632745  9323 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:46.724323 14055 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:46.724928  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1252) took 1009 ms. Trace:
W20260709 13:03:46.616899 14110 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.942s	user 0.019s	sys 0.000s
I20260709 13:03:46.725140 14055 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:46.725363 13751 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.946s	user 0.009s	sys 0.015s
W20260709 13:03:46.617362 14119 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.690s	user 0.010s	sys 0.009s
I20260709 13:03:46.725946 14036 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:46.647354 14109 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:46.725816 14055 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.647781  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2919) took 1435 ms. Trace:
I20260709 13:03:46.009583  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:46.727270  8583 rpcz_store.cc:276] 0709 13:03:45.212828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.011011 (+798183us) service_pool.cc:224] Handling call
0709 13:03:46.647717 (+636706us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:46.722371 14136 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:46.650105 14024 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.043s	sys 0.090s
I20260709 13:03:46.728438  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:46.728266 14136 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.648437 14027 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:46.728991 14024 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:46.729920 13743 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.643s	user 0.018s	sys 0.000s
I20260709 13:03:46.653913 13930 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:46.654331  8580 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:46.654872  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1014) took 1770 ms. Trace:
I20260709 13:03:46.730363 13743 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:46.654934  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2892) took 2155 ms. Trace:
I20260709 13:03:46.584211  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:46.732468 13743 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:46.732659  9325 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:46.731356  8582 rpcz_store.cc:276] 0709 13:03:44.499154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.499377 (+   223us) service_pool.cc:224] Handling call
0709 13:03:44.568644 (+ 69267us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:44.666541 (+ 97897us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:03:45.102251 (+435710us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.568499 (+466248us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:45.569110 (+   611us) log.cc:844] Serialized 52 byte log entry
0709 13:03:45.569259 (+   149us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:45.575304 (+  6045us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.578742 (+  3438us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.579231 (+   489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.579503 (+   272us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.654621 (+1075118us) raft_consensus.cc:1685] finished
0709 13:03:46.654653 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.654854 (+   201us) inbound_call.cc:177] Queueing success response
0709 13:03:46.656991 (+  2137us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b48002dffd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
Metrics: {"mutex_wait_us":9606,"prepare.queue_time_us":908355,"prepare.run_wall_time_us":221,"raft.queue_time_us":256800,"raft.run_cpu_time_us":11270,"raft.run_wall_time_us":1178149,"spinlock_wait_cycles":38053248,"thread_start_us":802932,"threads_started":2,"wal-append.queue_time_us":803600}
I20260709 13:03:46.657918 14140 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:46.733678  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.733750 14140 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.734388  8582 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
W20260709 13:03:46.734678 13992 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.981s	user 0.019s	sys 0.004s
I20260709 13:03:46.734980  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2913) took 1535 ms. Trace:
I20260709 13:03:46.735169 13992 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.731000  8584 rpcz_store.cc:276] 0709 13:03:44.884871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.197197 (+312326us) service_pool.cc:224] Handling call
0709 13:03:45.251124 (+ 53927us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.936905 (+685781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.654629 (+717724us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b540030c354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:46.654812 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1160376064}
I20260709 13:03:46.736729  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.720152  9327 rpcz_store.cc:276] 0709 13:03:45.109115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.114956 (+  5841us) service_pool.cc:224] Handling call
0709 13:03:45.115420 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.125200 (+  9780us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.125258 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.125999 (+   741us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.725642 (+599643us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:45.950527 (+224885us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.614028 (+663501us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.614160 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.614239 (+    79us) raft_consensus.cc:1685] finished
0709 13:03:46.614264 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.614519 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5392,"prepare.queue_time_us":924006,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":623736,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":26799,"thread_start_us":739133,"threads_started":2,"wal-append.queue_time_us":791684}
I20260709 13:03:46.729048  8583 raft_consensus.cc:2376] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11. Re-sending same reply.
I20260709 13:03:46.658768 14029 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.569754 14095 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:46.605674 13993 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:46.670003  8651 rpcz_store.cc:276] 0709 13:03:44.819465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.821835 (+  2370us) service_pool.cc:224] Handling call
0709 13:03:44.822319 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.838005 (+ 15686us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.838040 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.973447 (+135407us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b2c000b1fc8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:44.974265 (+   818us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.717989 (+743724us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:45.718032 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.718104 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.571579 (+853475us) raft_consensus.cc:1685] finished
0709 13:03:46.571599 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.571814 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15250,"raft.queue_time_us":15750,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":134,"spinlock_wait_cycles":14076800,"thread_start_us":615464,"threads_started":1,"wal-append.queue_time_us":851408}
W20260709 13:03:46.680230  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 94541.8us
I20260709 13:03:46.631510 14002 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:46.687019 14062 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:46.695724 14046 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.925s	user 0.009s	sys 0.023s
I20260709 13:03:46.696414 14034 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.696498  8579 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:46.717842  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:46.720854  8622 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.794s	user 0.005s	sys 0.018s
W20260709 13:03:46.821810  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 974146us
W20260709 13:03:46.606570 14000 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 2.111s	user 0.014s	sys 0.008s
W20260709 13:03:46.604233  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.729785 14024 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.656073 14144 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.722970  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:46.727670  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2923) took 1193 ms. Trace:
I20260709 13:03:46.733752  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 973) took 1179 ms. Trace:
W20260709 13:03:46.721133  8578 rpcz_store.cc:269] 0709 13:03:44.652566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.652732 (+   166us) service_pool.cc:224] Handling call
0709 13:03:46.720422 (+2067690us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b540002dd54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:46.720571 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4604962176}
I20260709 13:03:46.746104  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:46.750149  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.833777  8654 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:46.833969  8583 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:03:46.682969 14089 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.000s	sys 0.017s
I20260709 13:03:46.668795 13723 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:46.819813  9325 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:46.821686  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:46.800578  8730 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:46.735323  8582 rpcz_store.cc:276] 0709 13:03:45.199104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.733384 (+1534280us) service_pool.cc:224] Handling call
0709 13:03:46.734915 (+  1531us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:46.739521  8584 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:03:46.739223  9323 rpcz_store.cc:276] 0709 13:03:45.716038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.606435 (+890397us) service_pool.cc:224] Handling call
0709 13:03:46.632569 (+ 26134us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:46.724690 (+ 92121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.724869 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:46.823108  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2561) took 2183 ms. Trace:
I20260709 13:03:46.738225  9327 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:46.860224  8581 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:46.767180  8732 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:03:46.823647  8622 leader_election.cc:336] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.957s (SENT)
I20260709 13:03:46.834812  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2926) took 1165 ms. Trace:
I20260709 13:03:46.943276  9330 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 10: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:46.945312  8583 rpcz_store.cc:276] 0709 13:03:45.669702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.749849 (+1080147us) service_pool.cc:224] Handling call
0709 13:03:46.834731 (+ 84882us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:46.945539  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 121911us
I20260709 13:03:46.833145  9322 rpcz_store.cc:276] 0709 13:03:45.554237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.689361 (+135124us) service_pool.cc:224] Handling call
0709 13:03:45.715030 (+ 25669us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.715336 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.715377 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.721490 (+  6113us) log.cc:844] Serialized 26 byte log entry
0709 13:03:46.315461 (+593971us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:46.315501 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.315588 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.733541 (+417953us) raft_consensus.cc:1685] finished
0709 13:03:46.733562 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.733694 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":459707,"threads_started":1,"wal-append.queue_time_us":981486}
I20260709 13:03:46.832414  8576 rpcz_store.cc:276] 0709 13:03:45.534824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.951656 (+416832us) service_pool.cc:224] Handling call
0709 13:03:46.727609 (+775953us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:46.629715 14035 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.943414  9327 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:46.739049 13858 consensus_meta.cc:287] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.913s	user 0.010s	sys 0.011s
W20260709 13:03:46.894912  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 171995us
I20260709 13:03:46.958299  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.960929  8583 raft_consensus.cc:2368] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
W20260709 13:03:46.944501  8581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2922) took 1675 ms (client timeout 2072 ms). Trace:
W20260709 13:03:46.946854  8622 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.019s (SENT)
I20260709 13:03:46.944442  8727 rpcz_store.cc:276] 0709 13:03:44.638454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.654971 (+ 16517us) service_pool.cc:224] Handling call
0709 13:03:44.675921 (+ 20950us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.676263 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.676295 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.867003 (+190708us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b480018c258. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:44.869698 (+  2695us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b2c000b2cd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:44.870296 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.188209 (+317913us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:45.188248 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.188312 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.739217 (+1550905us) raft_consensus.cc:1685] finished
0709 13:03:46.739235 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.821369 (+ 82134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":95754,"raft.run_cpu_time_us":1599,"raft.run_wall_time_us":95114,"spinlock_wait_cycles":53833984,"thread_start_us":317595,"threads_started":1,"wal-append.queue_time_us":859297}
W20260709 13:03:46.961604  8581 rpcz_store.cc:269] 0709 13:03:45.269499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.641342 (+1371843us) service_pool.cc:224] Handling call
0709 13:03:46.944431 (+303089us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:46.961985  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:46.963200  8622 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.024s (SENT)
I20260709 13:03:46.963711  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:46.730749 14136 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.749651 14109 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:46.947698  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2931) took 1078 ms. Trace:
I20260709 13:03:46.966892  8584 rpcz_store.cc:276] 0709 13:03:45.869273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.736439 (+867166us) service_pool.cc:224] Handling call
0709 13:03:46.947636 (+211197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:46.759618  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:46.741003 14044 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.750074 14095 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.750386 13993 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.751199 14041 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.737339 13992 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.954226  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.954564  9330 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:46.954758  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.975585  8582 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:46.956117 14024 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.955760  9327 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.956698  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:46.747741 14050 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.758792 14028 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.821501 13948 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.985476  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2925) took 1326 ms. Trace:
I20260709 13:03:46.858531 14000 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.754896 14054 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:46.951941 14062 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:46.754776 13897 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:46.970566  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 217185us
I20260709 13:03:46.966898 14029 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:46.998010  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:46.998843  8652 raft_consensus.cc:2410] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:03:46.753508 14002 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:46.753750 14049 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.830503 14034 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:46.853837 14144 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:46.980484  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.980845  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.023568  8576 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:46.982959  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:47.024106  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:46.986119  8582 rpcz_store.cc:276] 0709 13:03:45.658654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.954478 (+1295824us) service_pool.cc:224] Handling call
0709 13:03:46.985343 (+ 30865us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:46.980511  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2932) took 1072 ms. Trace:
I20260709 13:03:46.987368  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:46.965093  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2933) took 1036 ms. Trace:
I20260709 13:03:46.949348 14014 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.964931  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:46.990701  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:46.991537  8656 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:46.994995  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:46.749486 14055 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.981122 14036 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:03:46.999120  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 359991us
I20260709 13:03:46.725827 13751 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.001170  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:46.824189 14046 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:47.001575  9294 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.744s (SENT)
I20260709 13:03:47.004096  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:47.010521  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.758921 14140 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:46.976796  9327 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:47.022140  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:46.999809 14005 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.588s	user 0.021s	sys 0.061s
I20260709 13:03:46.958498 13723 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:46.973245 14044 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:46.994678 14028 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:47.103948  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2934) took 1163 ms. Trace:
I20260709 13:03:47.104404  8655 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:47.121860  8576 rpcz_store.cc:276] 0709 13:03:45.940981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.958128 (+1017147us) service_pool.cc:224] Handling call
0709 13:03:47.103889 (+145761us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.108412  8657 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:03:47.103104  8578 rpcz_store.cc:276] 0709 13:03:45.908441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.833428 (+924987us) service_pool.cc:224] Handling call
0709 13:03:46.980448 (+147020us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.106478  8581 raft_consensus.cc:2376] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:03:47.103960  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:47.123385  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.107601  8651 raft_consensus.cc:2410] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 13:03:47.005014 14024 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:47.123767  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.124163  8576 raft_consensus.cc:2368] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:03:47.124580  8578 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:47.124300 14024 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.125032  8584 raft_consensus.cc:2410] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:03:47.125063  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.125437  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.126184  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2924) took 1575 ms. Trace:
I20260709 13:03:47.126368  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.126543 14024 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.024783 14021 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
I20260709 13:03:47.126654  8578 rpcz_store.cc:276] 0709 13:03:45.550699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.125158 (+1574459us) service_pool.cc:224] Handling call
0709 13:03:47.126130 (+   972us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.127619  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.127720  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.128741  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2935) took 1160 ms. Trace:
I20260709 13:03:47.112383 14029 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:47.111654 13751 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.028223 14027 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:47.130136 14027 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:47.130388  8578 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:47.113807  8577 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:47.023164 14049 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.134503  8659 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:46.972184 13858 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.115888 13993 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:47.000141 13897 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.995482 14000 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:46.995679 14054 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.120004 14030 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:46.988854 14050 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.122385 14144 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:47.105603  8583 rpcz_store.cc:276] 0709 13:03:45.929144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.958057 (+1028913us) service_pool.cc:224] Handling call
0709 13:03:46.964980 (+  6923us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:47.108229 14005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.692s	user 0.025s	sys 0.083s
I20260709 13:03:47.128675  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.131075  8581 rpcz_store.cc:276] 0709 13:03:45.968155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.962369 (+994214us) service_pool.cc:224] Handling call
0709 13:03:47.128672 (+166303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.138659 14123 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:46.995041 13948 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:47.138819  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 137267us
W20260709 13:03:47.142640  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 138659us
I20260709 13:03:47.130635 14027 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:47.144767  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.172078  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:47.172333  8582 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:47.146056  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1223) took 2351 ms. Trace:
I20260709 13:03:47.035359 14121 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:47.175976 14121 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.176764  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:47.176899  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:47.177709  9328 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:47.173674  9326 rpcz_store.cc:276] 0709 13:03:44.794968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.795208 (+   240us) service_pool.cc:224] Handling call
0709 13:03:44.806022 (+ 10814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.107079 (+301057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.145793 (+2038714us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b5400386b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:47.145980 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":664262016}
I20260709 13:03:47.138105 14134 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.154314  9331 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:47.150035  8729 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:47.110222 13692 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.122109 14028 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 13:03:47.163882  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.206583  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:47.146432 14040 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:47.166481  8659 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:47.168023  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.210843  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2558) took 2605 ms. Trace:
I20260709 13:03:47.170228  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.212834  8581 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:47.213877  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.144423 14036 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:47.178189  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.215356  8576 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:47.155071  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.176546  8727 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 13:03:47.182770  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 353343us
I20260709 13:03:47.141745 14014 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.208359  9327 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 13:03:47.220623  8733 raft_consensus.cc:2368] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:47.220691  9327 pending_rounds.cc:85] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 10
I20260709 13:03:47.221091  9327 pending_rounds.cc:107] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:03:47.221330  9327 raft_consensus.cc:2889] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:47.216881  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.207743  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.218403  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:47.225239  9323 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:47.219468  8578 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:47.225664  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.226289  8583 raft_consensus.cc:2376] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:03:47.227100  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:47.228348  8577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1017) took 2327 ms (client timeout 1869 ms). Trace:
I20260709 13:03:47.229882 14035 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.211233  8724 rpcz_store.cc:276] 0709 13:03:44.605014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.616628 (+ 11614us) service_pool.cc:224] Handling call
0709 13:03:44.650568 (+ 33940us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.117744 (+467176us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.117781 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.514058 (+396277us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b2c000b2cd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:45.514500 (+   442us) log.cc:844] Serialized 26 byte log entry
0709 13:03:46.637621 (+1123121us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:46.637668 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.637745 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.179981 (+542236us) raft_consensus.cc:1685] finished
0709 13:03:47.180043 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.210772 (+ 30729us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8107,"raft.queue_time_us":102466,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":18100,"spinlock_wait_cycles":11224448,"thread_start_us":1581240,"threads_started":2,"wal-append.queue_time_us":1615063}
I20260709 13:03:47.219966  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:47.228757  8577 rpcz_store.cc:269] 0709 13:03:44.901090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.907738 (+  6648us) service_pool.cc:224] Handling call
0709 13:03:47.228139 (+2320401us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54002c97d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:47.228260 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4925393152}
I20260709 13:03:47.234228  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.234956  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.236469  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.237558  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.238790  8583 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:47.239364  8579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1008) took 2390 ms (client timeout 1744 ms). Trace:
W20260709 13:03:47.243010  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.206907 14122 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.253809  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.256137  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.256743  8733 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
W20260709 13:03:47.247877  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 136979us
W20260709 13:03:47.239905  8579 rpcz_store.cc:269] 0709 13:03:44.847496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.852459 (+  4963us) service_pool.cc:224] Handling call
0709 13:03:47.237280 (+2384821us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54002d78d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:47.237374 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4185235840}
I20260709 13:03:47.247020  8581 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:47.250219  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:47.258090  9328 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:47.259584  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.260602  8581 raft_consensus.cc:2368] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12.
I20260709 13:03:47.251392  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:47.235877 14148 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:47.264153  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:47.265532  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.268229  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.269505  8582 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
W20260709 13:03:47.268648  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1219) took 2459 ms (client timeout 1687 ms). Trace:
I20260709 13:03:47.272089  8727 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:47.275254  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:47.283857  9323 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:47.275413  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.269448  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.286901  8578 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:47.281737  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:47.269274  9327 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:47.232884 14021 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:47.282472  9324 rpcz_store.cc:269] 0709 13:03:44.793672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.794125 (+   453us) service_pool.cc:224] Handling call
0709 13:03:47.236815 (+2442690us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54008b4054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:47.252498 (+ 15683us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4289923840}
I20260709 13:03:47.286084  8550 leader_election.cc:400] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
W20260709 13:03:47.287945  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 118556us
I20260709 13:03:47.297710  8550 leader_election.cc:403] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:03:47.276664  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 171708us
I20260709 13:03:47.291283  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.299458  8581 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 12 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:03:47.292840  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:47.300709  8733 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:47.301451  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.293988  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.255615 14123 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.302318  8577 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
W20260709 13:03:47.306129  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 278318us
I20260709 13:03:47.209929 13692 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.310459  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.312973  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.313838  8583 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:47.314633  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.311182  8578 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:47.268575 14148 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:47.319092  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43591.1us
I20260709 13:03:47.334472  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:47.209874 14134 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.359745  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1245) took 2273 ms. Trace:
I20260709 13:03:47.328729  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.332479  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.361111  8727 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:47.249786 13739 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 13:03:47.254583 14024 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 13:03:47.335386  8550 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:47.335817  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.336617 13751 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:47.339779 14109 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.297434 14028 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 13:03:47.349735 14099 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:47.351374 14155 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:47.354056 14151 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:03:47.354979 14035 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:47.364063  8733 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:47.360232  9331 rpcz_store.cc:276] 0709 13:03:45.086291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.086502 (+   211us) service_pool.cc:224] Handling call
0709 13:03:45.086827 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.178803 (+2091976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.349314 (+170511us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54007a99d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:47.359670 (+ 10356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4754104064}
I20260709 13:03:47.286378 13993 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:03:47.358323 14050 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.346685 13948 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:47.371718  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.367713  8581 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:47.368233  8583 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
W20260709 13:03:47.358385 14149 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:03:47.271665 14122 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:47.423666  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.329967 14144 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:03:47.503104  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.349757 14049 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:47.516077  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 214629us
I20260709 13:03:47.379762 14140 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:47.380620 14095 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.318405 14123 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.548046  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.510182 13723 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:47.471827 14099 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:47.471951 14021 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:47.369019 13751 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:47.584914  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 303244us
I20260709 13:03:47.585920  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.551348 14000 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.555563 13858 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:47.590305  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:47.557166  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 237472us
I20260709 13:03:47.511588 14034 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:47.504197 14035 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:47.682717 14162 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.683007 14035 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.552162  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.684737  8581 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:47.686305  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.687191  8581 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:47.688470  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.546857 14046 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.612668  8733 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:47.628086  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.630620 14099 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.691694  8583 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:47.692571 14099 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:47.692838  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.693851  8583 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:47.694816  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.695456  8583 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:47.641790 14040 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.696103 14040 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.631561 13948 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:47.655761  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.662215 13897 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:47.698523 14040 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:47.480376 14109 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.584062 14054 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:47.586861 14140 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:47.700074 14140 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:47.701062 14109 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:47.626837 14148 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.614430 14028 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:47.683532 14162 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.661285 13692 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.689224  8581 raft_consensus.cc:2376] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11. Re-sending same reply.
I20260709 13:03:47.714337  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.358824 14036 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.697708 13948 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.664646 14134 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.710309 14028 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:47.587453  8625 connection.cc:256] RPC call timeout handler was delayed by -1.9379514449999533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20260709 13:03:47.722961  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.557363 14027 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:47.628948 13751 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:47.624387 14021 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:47.752449  8580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1016) took 2855 ms (client timeout 2018 ms). Trace:
I20260709 13:03:47.719369  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.753449  8733 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:47.752996  8580 rpcz_store.cc:269] 0709 13:03:44.900728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.091351 (+190623us) service_pool.cc:224] Handling call
0709 13:03:47.752239 (+2660888us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540030b954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:47.752382 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3530344704}
I20260709 13:03:47.729203  8655 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:47.731227  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:47.754668  8656 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:47.756448 13948 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.738518  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:47.757297  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:47.738929  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:47.758092  8654 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:47.758740  8651 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:47.757716  8652 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:47.716239  8581 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:47.720394 14148 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:47.765266  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.782665  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.782883  8583 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 8.
I20260709 13:03:47.809828  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.787766  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.839761  8733 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:47.840148  8580 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:47.807032  8726 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:03:47.846186  8731 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:47.828202  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2954) took 1111 ms. Trace:
I20260709 13:03:47.847815  8581 rpcz_store.cc:276] 0709 13:03:46.716459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.782303 (+1065844us) service_pool.cc:224] Handling call
0709 13:03:47.828117 (+ 45814us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.849123  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.850170  8581 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:47.841600  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1009) took 1918 ms. Trace:
I20260709 13:03:47.862440  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:47.863222  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:47.852443  8726 rpcz_store.cc:276] 0709 13:03:45.923297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.568962 (+645665us) service_pool.cc:224] Handling call
0709 13:03:47.806066 (+1237104us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:47.806330 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.841256 (+ 34926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.841531 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55998848}
I20260709 13:03:47.880705  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.881875  8725 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:47.892668  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:47.893534  8580 raft_consensus.cc:2368] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
W20260709 13:03:47.810518 14158 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:03:47.724632 14134 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.154s
I20260709 13:03:47.761765 14154 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:47.802517 14156 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:47.863703 14062 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:47.996273  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.916399  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:47.999472  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 374191us
I20260709 13:03:47.999912  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.916852  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.001323  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:48.001554  8726 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:47.794425 14035 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:47.944531  8584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2917) took 2732 ms (client timeout 2899 ms). Trace:
W20260709 13:03:48.014320  8584 rpcz_store.cc:269] 0709 13:03:45.212168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.973071 (+1760903us) service_pool.cc:224] Handling call
0709 13:03:47.944269 (+971198us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54002ca1d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:47.944443 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233302400}
I20260709 13:03:48.002413  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:47.929790  8656 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
W20260709 13:03:48.003718  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 258543us
I20260709 13:03:48.016258  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.004814  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.088948  8581 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:48.074527  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1003) took 3607 ms (client timeout 2047 ms). Trace:
W20260709 13:03:48.075949  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2916) took 2864 ms (client timeout 2088 ms). Trace:
I20260709 13:03:48.092350  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2952) took 1376 ms. Trace:
I20260709 13:03:48.037220  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:48.011219 14029 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.012s	sys 0.018s
I20260709 13:03:48.017941  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.037591  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.093456  8726 raft_consensus.cc:2376] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13. Re-sending same reply.
W20260709 13:03:48.064465 14030 consensus_meta.cc:287] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.907s	user 0.011s	sys 0.001s
I20260709 13:03:48.073662 14121 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:48.092999  8581 rpcz_store.cc:276] 0709 13:03:46.715797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.004549 (+1288752us) service_pool.cc:224] Handling call
0709 13:03:48.092261 (+ 87712us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.939677 14162 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:47.937227 14123 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:48.096084 14005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.017s	sys 0.025s
I20260709 13:03:48.051659 14148 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:03:48.092553  8730 rpcz_store.cc:269] 0709 13:03:44.465623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.470397 (+  4774us) service_pool.cc:224] Handling call
0709 13:03:48.072722 (+3602325us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b5400203fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:48.072858 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5120864128}
W20260709 13:03:48.092592  8575 rpcz_store.cc:269] 0709 13:03:45.211822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.677099 (+465277us) service_pool.cc:224] Handling call
0709 13:03:48.075692 (+2398593us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b5400168ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:48.075855 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2230587264}
I20260709 13:03:48.016971  8725 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:48.080492 14057 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:48.084632 14143 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.262s	user 0.003s	sys 0.022s
I20260709 13:03:47.940554 14122 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:48.016938  8733 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:48.115346 14059 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:48.111753  8584 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 12 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:03:48.019429 14062 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:48.112504  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 111275us
I20260709 13:03:48.072211 14155 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:48.114025  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.041025  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.088543 14027 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:48.118860  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.039614 13639 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:03:48.114104 14029 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.013s	sys 0.018s
I20260709 13:03:48.119653 14148 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:47.929733 14154 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.950043 14021 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:47.943094 13930 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:48.129176  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.175534  8575 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
W20260709 13:03:48.104115  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 86319.1us
I20260709 13:03:48.182936  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.183904 14027 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:48.184763 14027 consensus_queue.cc:1243] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 13:03:48.186832  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.190112  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:48.191387 14116 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:48.191991 14116 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:48.192481 14116 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:48.195786 14062 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:48.195864  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:48.199086  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2555) took 3641 ms (client timeout 1583 ms). Trace:
W20260709 13:03:48.199509  8732 rpcz_store.cc:269] 0709 13:03:44.557554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.557695 (+   141us) service_pool.cc:224] Handling call
0709 13:03:48.198894 (+3641199us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b54004731d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:48.199019 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4989866240}
I20260709 13:03:48.017361 14166 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.105s
I20260709 13:03:48.199347  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:48.135267  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:48.202011 14062 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:48.202280 13739 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:48.202489 14062 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.203014 14062 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:48.202951 13739 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:48.137262  8581 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:48.206148  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2942) took 1493 ms. Trace:
W20260709 13:03:48.206451 14140 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.001s
I20260709 13:03:48.206513  8581 rpcz_store.cc:276] 0709 13:03:46.712769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.113745 (+1400976us) service_pool.cc:224] Handling call
0709 13:03:48.206083 (+ 92338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.207018 14140 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:48.139515  8726 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:48.033447 13948 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.208053 13948 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:48.071990 14170 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 13:03:48.136718 14155 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.209771 14155 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:48.211073  8726 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 13:03:48.212853 13948 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:48.154693  8583 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:48.138918 13639 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:48.213984  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2908) took 3100 ms. Trace:
I20260709 13:03:48.205166 13739 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:48.212924  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.214250 13639 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.215200  8727 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
W20260709 13:03:48.165711  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.153569 14021 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:48.215688 13639 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:48.198659 14148 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.216580 14148 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:48.217031  8696 leader_election.cc:336] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.028s (SENT)
I20260709 13:03:48.209227 14026 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:48.217950 14026 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:48.219033  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2815) took 1007 ms. Trace:
I20260709 13:03:48.219394 14140 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:48.220359 14026 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:48.159406 14154 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:48.222432 14044 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:48.225243  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.226096  8726 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:48.215968  8582 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:48.214681  8579 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 13:03:48.227082  8579 pending_rounds.cc:85] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 8
I20260709 13:03:48.214362  8583 rpcz_store.cc:276] 0709 13:03:45.113552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.142912 (+3029360us) service_pool.cc:224] Handling call
0709 13:03:48.154377 (+ 11465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:48.213718 (+ 59341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.213925 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.227459  8579 pending_rounds.cc:107] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 13:03:48.227746  8579 raft_consensus.cc:2889] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:48.219416  8731 rpcz_store.cc:276] 0709 13:03:47.211488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.220785 (+  9297us) service_pool.cc:224] Handling call
0709 13:03:48.218845 (+998060us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b54001e5fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:48.218974 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1254674688}
W20260709 13:03:48.229470  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.895889 14028 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.019690 14156 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.235978  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.235728  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:48.237049  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.241650  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2914) took 3016 ms. Trace:
I20260709 13:03:48.242851  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2912) took 3041 ms. Trace:
I20260709 13:03:48.243005  8582 rpcz_store.cc:276] 0709 13:03:45.211030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.215415 (+3004385us) service_pool.cc:224] Handling call
0709 13:03:48.215675 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.226855 (+ 11180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.227034 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.243561  8579 rpcz_store.cc:276] 0709 13:03:45.190489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.214074 (+3023585us) service_pool.cc:224] Handling call
0709 13:03:48.214360 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.228167 (+ 13807us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.228322 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.243749  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:48.249054  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 113911us
W20260709 13:03:48.202700 14144 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.617s	user 0.010s	sys 0.000s
I20260709 13:03:48.204506 13751 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.261833  8583 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:48.253238  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.252905  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:48.255234  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:48.263736  8658 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:03:48.264760  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:48.266569  8658 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:48.260646  8579 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:03:48.269562  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:48.272336  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.260842  8656 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:48.275238  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2918) took 3050 ms. Trace:
I20260709 13:03:48.275635  8583 rpcz_store.cc:276] 0709 13:03:45.212458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.228020 (+3015562us) service_pool.cc:224] Handling call
0709 13:03:48.254703 (+ 26683us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.262410 (+  7707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.262702 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.276432  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:48.268813  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2949) took 1554 ms. Trace:
I20260709 13:03:48.279057  8579 rpcz_store.cc:276] 0709 13:03:46.714895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.245924 (+1531029us) service_pool.cc:224] Handling call
0709 13:03:48.260288 (+ 14364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.268557 (+  8269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.268743 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.277326  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:48.281152  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:48.282472  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:48.283248  8657 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:48.284353  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:48.250392 14154 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:48.291589  8584 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:48.260766 14166 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:48.261243 14030 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:48.298799  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.299474  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.260126 14160 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:48.304209  8733 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:48.275868 14144 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:48.306663  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.302453  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.310118  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:48.311293  8583 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:48.304595  8579 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:48.311606  8731 raft_consensus.cc:2376] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13. Re-sending same reply.
I20260709 13:03:48.300885 14122 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 13:03:48.315117 14176 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:48.315686 14176 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.317713 14177 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:48.290858 14057 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:48.246966 13948 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:03:48.242049 13930 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.327392  8578 raft_consensus.cc:2368] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
I20260709 13:03:48.277798 14121 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 13:03:48.296034 14154 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:48.332190  8583 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 13:03:48.303845 14030 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.333722  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2953) took 1617 ms. Trace:
I20260709 13:03:48.330901  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2957) took 1613 ms. Trace:
I20260709 13:03:48.336709  8583 rpcz_store.cc:276] 0709 13:03:46.716162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.311930 (+1595768us) service_pool.cc:224] Handling call
0709 13:03:48.331887 (+ 19957us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.333491 (+  1604us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.333669 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.332152  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:48.338022  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.341071  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2563) took 3494 ms. Trace:
I20260709 13:03:48.341641  8729 rpcz_store.cc:276] 0709 13:03:44.846567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.915310 (+ 68743us) service_pool.cc:224] Handling call
0709 13:03:44.915903 (+   593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.179024 (+2263121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.340716 (+1161692us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b5400415d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:48.340987 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5137294080}
I20260709 13:03:48.337308  8579 rpcz_store.cc:276] 0709 13:03:46.717282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.312233 (+1594951us) service_pool.cc:224] Handling call
0709 13:03:48.330096 (+ 17863us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.330447 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.330490 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.330552 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:48.330608 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.330687 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.330841 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.331514 14005 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 13:03:48.346616 14155 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.348383  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.319443 14166 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:48.326161 14057 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.303402 14123 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 13:03:48.324754 14161 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:48.345031 14030 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:48.388880  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.391768  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.392788  8577 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:48.335835 14154 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:48.356647 14155 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:48.347940 13930 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:48.358048  8730 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:48.435597  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.436434  9323 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 9.
I20260709 13:03:48.436897  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.438114  9327 raft_consensus.cc:2410] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:03:48.361070  8583 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:48.362526  8625 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:48.362839 14173 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:03:48.363147 14177 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:48.363689  8579 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:48.319257 14160 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.462423  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:48.405812  8622 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.086s (SENT)
I20260709 13:03:48.463450  9322 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:48.463792  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:48.464852  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59069.5us
W20260709 13:03:48.465909  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:48.466512  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.468847  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.469830  8578 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:48.432083  9294 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:48.439882  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 77414.8us
I20260709 13:03:48.439790  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2959) took 1616 ms. Trace:
I20260709 13:03:48.471374  8583 rpcz_store.cc:276] 0709 13:03:46.823567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.337610 (+1514043us) service_pool.cc:224] Handling call
0709 13:03:48.360618 (+ 23008us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.439480 (+ 78862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.439717 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.417533  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.473764  8576 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13. Re-sending same reply.
I20260709 13:03:48.420657  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.474979  9331 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:48.475641  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:48.423264  9297 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:48.425055  8581 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 14. Ops: []
I20260709 13:03:48.435799  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.478578  9328 raft_consensus.cc:2410] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 13:03:48.451879  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:48.381721 14057 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:48.482180  9328 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:48.482303  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.483234  9330 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
W20260709 13:03:48.484387  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 61209us
I20260709 13:03:48.458354 14169 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:48.462486  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.462915  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2339) took 1477 ms. Trace:
I20260709 13:03:48.441867  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.473279  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:48.489130  8579 rpcz_store.cc:276] 0709 13:03:46.985931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.345023 (+1359092us) service_pool.cc:224] Handling call
0709 13:03:48.363464 (+ 18441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.462642 (+ 99178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.462853 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.476876  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:48.461252  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.405954 14136 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.494024  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.521329  8577 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:48.522023  8578 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:48.493140  9328 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:48.458701 14177 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:48.511897  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:48.416064 14014 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:48.530635  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.538523  8650 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:48.542078  8576 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:48.542351  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:48.549916  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 331570us
I20260709 13:03:48.536994  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:48.567107  8731 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:48.537148  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1015) took 3627 ms. Trace:
I20260709 13:03:48.568001  8584 rpcz_store.cc:276] 0709 13:03:44.896326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.139881 (+3243555us) service_pool.cc:224] Handling call
0709 13:03:48.150330 (+ 10449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.295762 (+145432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.518203 (+222441us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54000f82d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:48.518489 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":324286208}
I20260709 13:03:48.491878 14059 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:48.491425 14052 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:48.487973 14169 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.491636 14144 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:48.491765 14122 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:48.575765  9328 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:03:48.600263  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.600884  8579 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 13
W20260709 13:03:48.597546  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 336343us
I20260709 13:03:48.576532  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.535843  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.582643  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:48.582994  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:48.607185  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.594848  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.591967  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:48.600400  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.609617  8576 raft_consensus.cc:2368] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
I20260709 13:03:48.610617  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:48.563824 14121 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:48.618031  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.566252 14035 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:48.619444  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:48.608368  8696 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.099s (SENT)
I20260709 13:03:48.596133 14052 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.592978 14059 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:48.630782  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:48.622407  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.624722  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.627148  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.627565  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.634826  8584 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:48.564625 14136 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:48.617599 14014 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:48.598814 14144 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.599820 14169 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:48.628378 14170 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:48.655308  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2920) took 3442 ms. Trace:
I20260709 13:03:48.655716  8584 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:48.637106  8583 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:48.600059 14122 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:48.641237  8577 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:48.642719  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2962) took 1669 ms. Trace:
I20260709 13:03:48.625926 13639 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:48.617437 14121 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.621232 14041 raft_consensus.cc:2764] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:03:48.618548 13751 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:48.627974  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.636034 14059 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:48.642486 14014 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:48.630774 14052 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:48.659313  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:48.680088  9323 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:48.661054  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2348) took 1405 ms. Trace:
I20260709 13:03:48.681046  8584 rpcz_store.cc:276] 0709 13:03:47.255425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.635533 (+1380108us) service_pool.cc:224] Handling call
0709 13:03:48.650371 (+ 14838us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.660723 (+ 10352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.660981 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:48.657465  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3031) took 1662 ms (client timeout 1641 ms). Trace:
W20260709 13:03:48.682281  8659 rpcz_store.cc:269] 0709 13:03:46.986933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.987126 (+   193us) service_pool.cc:224] Handling call
0709 13:03:48.648860 (+1661734us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b54001310d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:48.648977 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70325632}
I20260709 13:03:48.664881  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2341) took 1658 ms. Trace:
I20260709 13:03:48.667158  8578 raft_consensus.cc:2368] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
I20260709 13:03:48.683583  8577 rpcz_store.cc:276] 0709 13:03:47.006219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.621326 (+1615107us) service_pool.cc:224] Handling call
0709 13:03:48.641013 (+ 19687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.664639 (+ 23626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.664828 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.618583 14123 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 13:03:48.655997  8582 rpcz_store.cc:276] 0709 13:03:45.213169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.244583 (+3031414us) service_pool.cc:224] Handling call
0709 13:03:48.274205 (+ 29622us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.654809 (+380604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.654858 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.654922 (+    64us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:48.654982 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.655068 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.655233 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37366,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":11125,"thread_start_us":353264,"threads_started":1}
I20260709 13:03:48.643356 14026 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:48.651160 14144 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:48.656960  8580 rpcz_store.cc:276] 0709 13:03:46.973723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.339370 (+1365647us) service_pool.cc:224] Handling call
0709 13:03:48.339731 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.642273 (+302542us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.642302 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.642352 (+    50us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:48.642406 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.642496 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.642649 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46,"raft.queue_time_us":15860,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":62,"thread_start_us":302062,"threads_started":1}
I20260709 13:03:48.676430  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.662992 14121 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:48.633673 13692 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 13:03:48.646893 14176 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:48.684963  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.699937  8577 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:48.701150  8577 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:48.632575 14035 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.659052 14107 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:03:48.693282  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 207323us
I20260709 13:03:48.657470 14122 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:48.677644  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 69323.9us
W20260709 13:03:48.711566  8622 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.647s (SENT)
I20260709 13:03:48.714278  9297 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:48.714430  8696 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.166s (SENT)
W20260709 13:03:48.714728  8622 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.842s (SENT)
I20260709 13:03:48.715072  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:48.715297  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:48.694164 14179 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.296s	user 0.001s	sys 0.000s
I20260709 13:03:48.722671  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.706866  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.723321  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.723831  8580 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:48.670799 13930 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:48.702090  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.721240  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2349) took 1420 ms. Trace:
I20260709 13:03:48.730630  8577 rpcz_store.cc:276] 0709 13:03:47.286800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.700648 (+1413848us) service_pool.cc:224] Handling call
0709 13:03:48.700976 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.706797 (+  5821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.707018 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.732128  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.690865 14014 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:48.703428  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.732720  8583 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:48.737854  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.723403  8582 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
I20260709 13:03:48.693432 14049 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:48.731796  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.747995  8577 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 7. Current term is 8. Ops: []
W20260709 13:03:48.739698  8696 connection.cc:256] RPC call timeout handler was delayed by -3.1510593740000559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23060000000000003s for a response
I20260709 13:03:48.748623  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2911) took 3593 ms. Trace:
I20260709 13:03:48.749212  8581 rpcz_store.cc:276] 0709 13:03:45.155609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.207367 (+3051758us) service_pool.cc:224] Handling call
0709 13:03:48.424805 (+217438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.477705 (+ 52900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.748410 (+270705us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54004407d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:48.748554 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":499283328}
W20260709 13:03:48.748974  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.771s (SENT)
W20260709 13:03:48.757368  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.077s (SENT)
W20260709 13:03:48.758227  8696 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.088s (SENT)
I20260709 13:03:48.758421  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2351) took 1423 ms. Trace:
W20260709 13:03:48.758966  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.113s (SENT)
I20260709 13:03:48.759505  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.758905  8583 rpcz_store.cc:276] 0709 13:03:47.315888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.738399 (+1422511us) service_pool.cc:224] Handling call
0709 13:03:48.738742 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.739083 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.739123 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.739180 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:48.739226 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.739276 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.739368 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:48.725389 14148 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.508s	user 0.012s	sys 0.000s
I20260709 13:03:48.762756  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2359) took 1162 ms. Trace:
I20260709 13:03:48.763198  8577 rpcz_store.cc:276] 0709 13:03:47.585745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.747591 (+1161846us) service_pool.cc:224] Handling call
0709 13:03:48.747826 (+   235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.748358 (+   532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.748511 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:48.765520  8696 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.185s (SENT)
I20260709 13:03:48.766000  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.767136  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.717526 14156 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:48.777663  8652 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:48.731563 14026 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:48.787639  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.747012 14049 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.728356 14190 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:48.839888  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.883267  8583 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:48.846300  8578 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:48.884359  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1025) took 1243 ms. Trace:
I20260709 13:03:48.884891  8578 rpcz_store.cc:276] 0709 13:03:47.641019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.787261 (+1146242us) service_pool.cc:224] Handling call
0709 13:03:48.845839 (+ 58578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.883996 (+ 38157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.884288 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:48.879614  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 114086us
W20260709 13:03:48.886305  8696 connection.cc:256] RPC call timeout handler was delayed by -2.929975708000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20450000000000002s for a response
W20260709 13:03:48.886821  8696 connection.cc:256] RPC call timeout handler was delayed by -2.9056757080001034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
W20260709 13:03:48.887300  8696 connection.cc:256] RPC call timeout handler was delayed by -2.6826994860000468s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33540000000000003s for a response
W20260709 13:03:48.887756  8696 connection.cc:256] RPC call timeout handler was delayed by -2.6715949160001173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20260709 13:03:48.888226  8696 connection.cc:256] RPC call timeout handler was delayed by -2.5800494350000918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260709 13:03:48.888753  8696 connection.cc:256] RPC call timeout handler was delayed by -2.54912180000008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
W20260709 13:03:48.889252  8696 connection.cc:256] RPC call timeout handler was delayed by -2.0899870969999483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20260709 13:03:48.889834  8696 connection.cc:256] RPC call timeout handler was delayed by -2.0625098250000065s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2899s for a response
I20260709 13:03:48.810454  8581 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:48.812258  8582 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:48.893329  8577 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:48.896534  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2361) took 1185 ms. Trace:
W20260709 13:03:48.897572  8696 connection.cc:256] RPC call timeout handler was delayed by -1.7770757080000976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3326s for a response
W20260709 13:03:48.899019  8696 connection.cc:256] RPC call timeout handler was delayed by -1.1962073450000617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
I20260709 13:03:48.897056  8577 rpcz_store.cc:276] 0709 13:03:47.710952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.877375 (+1166423us) service_pool.cc:224] Handling call
0709 13:03:48.881141 (+  3766us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.896254 (+ 15113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.896463 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:48.843400 14179 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
W20260709 13:03:48.799428 14181 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260709 13:03:48.766974 14160 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:48.893821 14180 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
I20260709 13:03:48.786387 14041 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:48.967945 14026 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:48.907065  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2352) took 1562 ms. Trace:
I20260709 13:03:48.878959 14049 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:48.901973  8577 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:48.923468  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:48.975219  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 392618us
I20260709 13:03:48.976576  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:48.980000 14099 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:48.982194  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2363) took 1258 ms. Trace:
I20260709 13:03:48.982949  8577 rpcz_store.cc:276] 0709 13:03:47.723826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.901312 (+1177486us) service_pool.cc:224] Handling call
0709 13:03:48.901743 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.981926 (+ 80183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.982127 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.984364  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:48.986248  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:48.987710  8577 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:48.991274  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1016) took 1730 ms. Trace:
I20260709 13:03:48.991722  8730 rpcz_store.cc:276] 0709 13:03:47.261043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.122798 (+861755us) service_pool.cc:224] Handling call
0709 13:03:48.123159 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.433251 (+310092us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.991038 (+557787us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54000ea454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:48.991215 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":539160448}
I20260709 13:03:48.963770 14174 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:48.990545  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1032) took 1235 ms. Trace:
I20260709 13:03:48.995743  8577 rpcz_store.cc:276] 0709 13:03:47.755591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.988778 (+1233187us) service_pool.cc:224] Handling call
0709 13:03:48.989114 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.989304 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.989335 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.989382 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:48.989450 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.989511 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.990479 (+   968us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:48.834203 14163 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
W20260709 13:03:48.962478  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44891.9us
I20260709 13:03:48.904018  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.010241  8578 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:03:49.010834  8696 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.005s (SENT)
W20260709 13:03:49.012068  8696 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.045s (SENT)
I20260709 13:03:49.012557  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:48.970353  8582 rpcz_store.cc:276] 0709 13:03:47.333620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.740167 (+1406547us) service_pool.cc:224] Handling call
0709 13:03:48.788811 (+ 48644us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.895653 (+106842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.895947 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.907830  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2360) took 1276 ms. Trace:
I20260709 13:03:48.996955  8580 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:03:49.014075  8581 rpcz_store.cc:276] 0709 13:03:47.619379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.750343 (+1130964us) service_pool.cc:224] Handling call
0709 13:03:48.810176 (+ 59833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.895075 (+ 84899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.895357 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:49.017349  8696 leader_election.cc:336] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.072s (SENT)
W20260709 13:03:48.990639 14202 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:03:48.940140 14170 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:48.966801 14201 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:48.977860 14197 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:49.004148  9322 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.540s	user 0.009s	sys 0.004s
W20260709 13:03:49.005026  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 260204us
I20260709 13:03:49.066551  9322 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:03:49.066779  8622 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.113s (SENT)
W20260709 13:03:49.067833  8622 connection.cc:256] RPC call timeout handler was delayed by -1.4060197729999118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1751s for a response
W20260709 13:03:49.072185  8622 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.751s (SENT)
I20260709 13:03:49.086385  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:48.966097 14182 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.324s	user 0.001s	sys 0.000s
I20260709 13:03:48.982689 14148 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:49.022212  8696 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.117s (SENT)
I20260709 13:03:49.022571  8580 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
W20260709 13:03:49.022948  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 299700us
W20260709 13:03:49.092324  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70162.5us
W20260709 13:03:49.091879  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.007s	sys 0.015s
W20260709 13:03:49.022867  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44016.3us
I20260709 13:03:48.997970  8583 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:49.057636  8578 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:49.062783  8581 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:03:49.085870  8582 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:03:49.087678  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.008575  8577 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:49.056952 14026 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:49.092185  8580 pending_rounds.cc:85] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 8
I20260709 13:03:49.100754  8580 pending_rounds.cc:107] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 13:03:49.101119  8580 raft_consensus.cc:2889] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:48.981245 14014 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:48.402145 14161 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.098435  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2365) took 1316 ms. Trace:
W20260709 13:03:49.093281  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.007s	sys 0.015s
W20260709 13:03:49.093477  8696 leader_election.cc:336] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 2.306s (SENT)
W20260709 13:03:49.106227  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 629394us
I20260709 13:03:49.106328  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:49.106050  8577 rpcz_store.cc:276] 0709 13:03:47.782496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.999144 (+1216648us) service_pool.cc:224] Handling call
0709 13:03:48.999531 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.098127 (+ 98596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.098361 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:49.103241  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.007s	sys 0.012s
I20260709 13:03:49.107115  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:49.093220  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:49.096819 14183 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.447s	user 0.001s	sys 0.000s
W20260709 13:03:49.151002  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57559.2us
W20260709 13:03:49.147796  8549 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.7.78.126:44983, user_credentials={real_user=slave}} blocked reactor thread for 43823.5us
I20260709 13:03:49.165779  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.065706 14201 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:49.067294 14136 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:49.155786  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 62647us
I20260709 13:03:49.169416  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.170159  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.171142  8577 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:49.186242  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:49.186499 14203 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 13:03:49.186817 14197 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.186993  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.199661  8582 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:49.192435  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:49.107568 13739 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:49.191938  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.213835  8577 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:49.214123 14026 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.214586 14026 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:49.168146 14201 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:49.221907 14186 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
I20260709 13:03:49.223517 14161 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:49.224526 14193 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.203s	user 0.000s	sys 0.000s
I20260709 13:03:49.092901 14213 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:49.225538 14213 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:03:49.229765 14188 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.351s	user 0.000s	sys 0.001s
I20260709 13:03:49.231230 14178 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:49.231827 14178 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:49.235597 14160 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:49.236157 14160 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:49.240594 14196 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.268s	user 0.000s	sys 0.000s
I20260709 13:03:49.224603 14161 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:49.241854 14161 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.242560 14156 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:49.242933 14156 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:49.243821 14156 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:48.549491 14166 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.167184 14055 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:49.255342  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:49.255856  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.256810  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:49.202015  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.205029 14217 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:49.268010 14217 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:49.205338  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:49.269004 14174 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:49.269268  8582 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:49.269505 14174 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:49.270262 14208 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:03:49.205515 14197 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.274786  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1022) took 1569 ms. Trace:
I20260709 13:03:49.199536  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.208283  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2910) took 4060 ms. Trace:
I20260709 13:03:49.199401 13692 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20260709 13:03:49.276592 13692 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:49.276599  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:49.277331 13692 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:49.279455 13692 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:49.279722  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2340) took 2285 ms. Trace:
I20260709 13:03:49.268036  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 905) took 1016 ms. Trace:
I20260709 13:03:49.280081  8576 rpcz_store.cc:276] 0709 13:03:46.994523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.615687 (+1621164us) service_pool.cc:224] Handling call
0709 13:03:48.631666 (+ 15979us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.279254 (+647588us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b1800384100. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f1409c0e060 00007f1409c0ebfb 00007f1409c1964b
0709 13:03:49.279388 (+   134us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:49.279412 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:49.279459 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:49.279492 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.279560 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.279669 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31366,"raft.queue_time_us":87709,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":5402,"spinlock_wait_cycles":10384128,"thread_start_us":152938,"threads_started":1}
I20260709 13:03:49.195753 14035 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:49.281139  8653 rpcz_store.cc:276] 0709 13:03:48.251906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.253453 (+  1547us) service_pool.cc:224] Handling call
0709 13:03:48.282070 (+ 28617us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.348377 (+ 66307us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:48.357674 (+  9297us) log.cc:844] Serialized 14 byte log entry
0709 13:03:48.987138 (+629464us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.987199 (+    61us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:48.987247 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.267797 (+280550us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:49.267842 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.267981 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3320,"prepare.queue_time_us":14803,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":254901,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":55598,"spinlock_wait_cycles":152004480,"thread_start_us":628195,"threads_started":2,"wal-append.queue_time_us":485971}
I20260709 13:03:49.284209 14014 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:49.287142 14062 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.084s	user 0.007s	sys 0.006s
W20260709 13:03:49.207811  8579 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.591s	user 0.006s	sys 0.001s
I20260709 13:03:49.214788  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.289186  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.289985  8577 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:49.290084 14223 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:49.292258  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.250411  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.252378  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.293551  8652 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:49.293757  8579 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:49.237211 14160 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:49.256954 14021 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.041s	user 0.008s	sys 0.001s
I20260709 13:03:48.631348 14177 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.257063  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:49.257620  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48115.9us
W20260709 13:03:49.257980 14212 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:03:49.312613  8730 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:49.258427 14170 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:49.258939  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.275997  8575 rpcz_store.cc:276] 0709 13:03:45.148265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.176249 (+3027984us) service_pool.cc:224] Handling call
0709 13:03:48.176747 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:48.177076 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.177117 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:48.177764 (+   647us) log.cc:844] Serialized 26 byte log entry
0709 13:03:48.480987 (+303223us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:48.481034 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.481110 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:49.208077 (+726967us) raft_consensus.cc:1685] finished
0709 13:03:49.208095 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.208227 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":302787,"threads_started":1,"wal-append.queue_time_us":422171}
I20260709 13:03:49.313490  8656 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:49.259467  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2569) took 1038 ms. Trace:
I20260709 13:03:49.314378  8727 rpcz_store.cc:276] 0709 13:03:48.221298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.222211 (+   913us) service_pool.cc:224] Handling call
0709 13:03:48.222612 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:48.222965 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.222995 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:48.223480 (+   485us) log.cc:844] Serialized 26 byte log entry
0709 13:03:48.686187 (+462707us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:48.686251 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.686349 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:49.259245 (+572896us) raft_consensus.cc:1685] finished
0709 13:03:49.259266 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.259406 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":156,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":462083,"threads_started":1,"wal-append.queue_time_us":611128}
W20260709 13:03:49.260303 14154 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.800s	user 0.011s	sys 0.000s
I20260709 13:03:49.217238 14026 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:49.260461  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.261891 14099 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:49.242213 14161 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:49.264015 14057 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.752s	user 0.006s	sys 0.003s
I20260709 13:03:49.266580 14201 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.275579 14197 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:49.275835  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:49.377571  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.232851 14178 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.294574  8583 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:49.297224  8582 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:49.297971  8580 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:49.275188  8725 rpcz_store.cc:276] 0709 13:03:47.705455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.133568 (+428113us) service_pool.cc:224] Handling call
0709 13:03:48.134038 (+   470us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:48.139876 (+  5838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.139897 (+    21us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:48.140384 (+   487us) log.cc:844] Serialized 52 byte log entry
0709 13:03:48.308668 (+168284us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:48.654785 (+346117us) log.cc:844] Serialized 14 byte log entry
0709 13:03:48.655429 (+   644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.656011 (+   582us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:49.274563 (+618552us) raft_consensus.cc:1685] finished
0709 13:03:49.274583 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.274729 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3435,"prepare.queue_time_us":385312,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":616,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":12291,"thread_start_us":483946,"threads_started":2,"wal-append.queue_time_us":454849}
I20260709 13:03:49.304891  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:49.381946  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.309152  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:49.382723  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 73667.2us
I20260709 13:03:49.383639  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:49.316974  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60232.2us
I20260709 13:03:49.318470  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:49.318053  8478 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 10, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 10 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:49.318898 13739 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 13:03:49.269013 14217 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.319710 14160 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:49.308223 14055 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:49.324029  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 908) took 1068 ms. Trace:
I20260709 13:03:49.325162  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.325693 14194 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:49.325726 14155 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:49.326937 13930 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:49.327559 14200 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:03:49.329046 14122 consensus_meta.cc:287] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.593s	user 0.006s	sys 0.002s
I20260709 13:03:49.330751  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:49.213150 14059 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.519s	user 0.005s	sys 0.000s
I20260709 13:03:49.399308 13639 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.380646 14021 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.324076  8652 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:49.156283  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.012s	sys 0.028s
W20260709 13:03:49.350369  8729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.013s	sys 0.035s
I20260709 13:03:49.371613  8581 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:49.438444  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 946141us
W20260709 13:03:49.342813 14116 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.148s	user 0.001s	sys 0.013s
I20260709 13:03:49.287602 14062 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.378486  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:49.451691  8576 raft_consensus.cc:2376] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16. Re-sending same reply.
I20260709 13:03:49.311880  8577 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:03:49.400147  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.412932  8650 rpcz_store.cc:276] 0709 13:03:48.255871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.255998 (+   127us) service_pool.cc:224] Handling call
0709 13:03:49.323841 (+1067843us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540005a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:49.323969 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":582765440}
I20260709 13:03:49.452688  8580 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:49.413192  8657 raft_consensus.cc:2376] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:03:49.412583 14099 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:49.422932  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.453981  8728 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 13:03:49.423710  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45025.4us
I20260709 13:03:49.426011  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:49.342345 14213 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 13:03:49.399379 14005 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
W20260709 13:03:49.343180  8651 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.021s	sys 0.025s
W20260709 13:03:49.462265  8651 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.022s	sys 0.025s
I20260709 13:03:49.348788 13692 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
I20260709 13:03:49.341719 14154 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:49.443656  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.470096  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:49.443679  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:49.445333  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:49.412840 14055 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:49.476505  9329 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
W20260709 13:03:49.460996 14121 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.739s	user 0.008s	sys 0.002s
I20260709 13:03:49.450554  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:49.424860 14041 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:49.455107  8583 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
W20260709 13:03:49.400126 14052 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.700s	user 0.009s	sys 0.000s
I20260709 13:03:49.483799  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:49.438621 14169 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.775s	user 0.008s	sys 0.000s
I20260709 13:03:49.380873 14057 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.432551 14122 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:49.447642 14219 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 13:03:49.470857 14028 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:49.438493 14107 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 13:03:49.447438 13639 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:49.430362 14030 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.043s	user 0.009s	sys 0.004s
I20260709 13:03:49.452395 14116 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:49.478614  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 311285us
I20260709 13:03:49.455062 14099 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:49.486320  8729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.991s	user 0.014s	sys 0.035s
I20260709 13:03:49.399601 14221 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.081s
I20260709 13:03:49.489354  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:49.517283  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 410229us
I20260709 13:03:49.494889  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.496626  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.498247  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.475602 14055 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:49.475543 14210 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:49.447355 14021 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.438355 14059 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.369634 14035 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 13:03:49.519923  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.518177  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.494652  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:49.456593 14206 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 13:03:49.597009 14218 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:49.597486 14218 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:49.487246 14121 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.600128  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 985) took 1157 ms. Trace:
I20260709 13:03:49.605263  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1244) took 1014 ms. Trace:
I20260709 13:03:49.605763  9329 rpcz_store.cc:276] 0709 13:03:48.590911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.689018 (+ 98107us) service_pool.cc:224] Handling call
0709 13:03:48.689328 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:49.477043 (+787715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.604911 (+127868us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b54006c2d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:49.605191 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1809533312}
I20260709 13:03:49.611877  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.526551  8584 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 11 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:49.526907  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.487807 14052 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.554565 14166 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.522477  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:49.500648 13930 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:49.595098  8727 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:49.528434  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:49.659759  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2358) took 2076 ms. Trace:
W20260709 13:03:49.660450  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 132119us
I20260709 13:03:49.660269  8584 rpcz_store.cc:276] 0709 13:03:47.583059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.746970 (+1163911us) service_pool.cc:224] Handling call
0709 13:03:48.747269 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.614792 (+867523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.659492 (+ 44700us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b5400179154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:49.659693 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1791571200}
I20260709 13:03:49.661262  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:49.600623  9331 rpcz_store.cc:276] 0709 13:03:48.443191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.476123 (+ 32932us) service_pool.cc:224] Handling call
0709 13:03:48.476531 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.194720 (+718189us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:49.204034 (+  9314us) log.cc:844] Serialized 14 byte log entry
0709 13:03:49.487244 (+283210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:49.487303 (+    59us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:49.487363 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.599840 (+112477us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b5400519354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:49.599891 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.600046 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":9570,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":104038,"spinlock_wait_cycles":1651077888,"thread_start_us":282892,"threads_started":1}
I20260709 13:03:49.614524  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.613353  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.615749  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.639829  8657 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:49.662931  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.663995  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.618736  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2580) took 1141 ms. Trace:
W20260709 13:03:49.651198  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 191241us
I20260709 13:03:49.631892  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.708042  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.633317  9326 raft_consensus.cc:2368] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12.
I20260709 13:03:49.633910  8726 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:49.633613  9324 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:49.664323  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 141895us
I20260709 13:03:49.645988 14216 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:49.351537  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:49.555938  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.556293  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.668387  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:49.492436 14169 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.555613 14099 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:49.673101  8478 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 6 to 9, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 9 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:49.714342  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.616783 14035 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:03:49.646099 14049 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.657s	user 0.006s	sys 0.004s
I20260709 13:03:49.633045 14059 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:49.565696  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.731932  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.732571  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.626951 14177 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:49.738541  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.622816 14178 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:49.684500  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 299883us
I20260709 13:03:49.718964  9331 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:49.705893  8724 rpcz_store.cc:276] 0709 13:03:48.477515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.479578 (+  2063us) service_pool.cc:224] Handling call
0709 13:03:48.532984 (+ 53406us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:48.543940 (+ 10956us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.543998 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:48.574128 (+ 30130us) log.cc:844] Serialized 52 byte log entry
0709 13:03:49.104819 (+530691us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:49.104896 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.104993 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:49.618499 (+513506us) raft_consensus.cc:1685] finished
0709 13:03:49.618522 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.618676 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10544,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":29922,"thread_start_us":480875,"threads_started":1,"wal-append.queue_time_us":686959}
I20260709 13:03:49.719846  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:49.721639  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.723209  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.624127 13639 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:49.632738 14052 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:49.621704 14057 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:49.633685 14154 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:49.730504  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.626045 14062 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:49.627547 14122 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:49.715157 14216 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:49.735141  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.621645 14121 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:49.631575 14021 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:49.737470  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 267458us
I20260709 13:03:49.747830  8653 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:49.660108 13930 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:49.750846  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:49.751396  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.524048 14030 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.728572 14035 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:49.809602  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.811244  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.595304 14174 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:49.907267  8477 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 304) took 39767 ms (client timeout 15000 ms). Trace:
I20260709 13:03:49.822566  9323 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:49.918310  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.813122  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 988) took 1318 ms. Trace:
I20260709 13:03:49.919934  9324 rpcz_store.cc:276] 0709 13:03:48.495254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.495429 (+   175us) service_pool.cc:224] Handling call
0709 13:03:48.541916 (+ 46487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.709302 (+1167386us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.812838 (+103536us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540037a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:49.813056 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2509936000}
I20260709 13:03:49.824491  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.705832 14223 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:49.922470 14223 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.737824 14218 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:49.847468  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.927237 14169 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:49.738294 14049 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.930727 14049 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:49.932823 14156 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:49.936718 14030 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:49.937532 14226 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:03:49.939311 14225 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.265s	user 0.000s	sys 0.000s
I20260709 13:03:49.940601  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1242) took 1350 ms. Trace:
I20260709 13:03:49.940956  9330 rpcz_store.cc:276] 0709 13:03:48.590213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.590419 (+   206us) service_pool.cc:224] Handling call
0709 13:03:48.607912 (+ 17493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:49.094029 (+486117us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:49.094093 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:49.094817 (+   724us) log.cc:844] Serialized 26 byte log entry
0709 13:03:49.516298 (+421481us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:49.516350 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.516440 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:49.940343 (+423903us) raft_consensus.cc:1685] finished
0709 13:03:49.940373 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.940547 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":272495,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":68972,"thread_start_us":906697,"threads_started":2,"wal-append.queue_time_us":552272}
I20260709 13:03:49.876013 14057 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
W20260709 13:03:49.950120 14233 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 13:03:49.876299 14154 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:49.738108 14201 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:49.954331  9329 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:49.871531  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:49.956779  9328 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:49.873382  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 461889us
I20260709 13:03:49.670861 14166 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.633687 14210 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:49.809046  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:49.811513 14059 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:49.893734  9325 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:49.734253 14227 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:49.968470  9296 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.014s	sys 0.036s
I20260709 13:03:49.897995  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:49.895162  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.899055  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:49.970294  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.902904  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:49.907624 14174 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:49.620517 14123 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.115s
I20260709 13:03:49.909747  8478 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 10, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 10 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:49.913693  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:49.914674  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:49.921555  9324 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:49.839710  8652 raft_consensus.cc:2368] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:49.880336  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.888064  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:49.888716  8577 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:49.961128  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:49.968907  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 160001us
I20260709 13:03:49.901762  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:50.023448  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60535.9us
I20260709 13:03:49.893639 13639 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:49.879338  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.985948  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:50.026181  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:50.026587  9328 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:49.984933  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:49.977013  8730 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:49.897845 13930 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:50.028748  8732 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:49.970851  9296 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.015s	sys 0.036s
I20260709 13:03:50.036937  8548 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:50.038298  9331 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:50.034319  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:50.040606  8725 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:50.043546  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:50.044883  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.046293  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.047832  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.048328  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:50.049180  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.051757  9323 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:50.055763  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.057349  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.058326  8653 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:50.060099  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.060709  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:50.061676  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.046438  8478 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 9, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 9 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:50.063275  8651 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.018s	sys 0.078s
W20260709 13:03:50.063782  8651 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.018s	sys 0.079s
I20260709 13:03:50.063246  8478 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 8, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 8 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:03:50.072856  8729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.035s	sys 0.076s
W20260709 13:03:50.073460  8729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.036s	sys 0.076s
W20260709 13:03:50.023736 14041 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.027s	sys 0.036s
I20260709 13:03:50.086418  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:50.087563  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2579) took 1610 ms. Trace:
I20260709 13:03:50.090979  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:49.976845 14059 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:50.087913  8729 rpcz_store.cc:276] 0709 13:03:48.477107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.478180 (+  1073us) service_pool.cc:224] Handling call
0709 13:03:48.478575 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.478844 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:49.520971 (+1042127us) log.cc:844] Serialized 14 byte log entry
0709 13:03:50.087106 (+566135us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:50.087157 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:50.087209 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.087277 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:50.087506 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13104,"prepare.queue_time_us":508151,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"thread_start_us":1528509,"threads_started":2}
I20260709 13:03:50.095834  8724 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:03:50.080935  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 119934us
I20260709 13:03:50.082794  8697 leader_election.cc:400] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12.
I20260709 13:03:50.078908  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:50.084357 14041 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.027s	sys 0.036s
I20260709 13:03:50.060027 14154 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:49.969193 14014 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.118s
I20260709 13:03:50.097791  8697 leader_election.cc:403] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:03:50.098043  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:50.164064  8652 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:50.097714  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.150408  9326 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:50.167335  8653 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:50.169327  8653 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:50.171728  8653 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:50.176679  8652 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:50.177139  8725 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
W20260709 13:03:50.194234  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 487855us
I20260709 13:03:50.193507  8478 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 8, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 8 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:50.090605 14174 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:50.090435 14162 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:50.086386 13948 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.059659 13639 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:50.205066  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 122458us
W20260709 13:03:50.205134  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 50217.2us
W20260709 13:03:50.173723 14028 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.658s	user 0.007s	sys 0.003s
I20260709 13:03:50.082056 13739 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.114s
I20260709 13:03:50.199112  9329 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:50.220865  8478 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 8, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 8 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:50.222272  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:50.098481 14041 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:50.195729 14144 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.479s	user 0.013s	sys 0.000s
I20260709 13:03:50.117182 14154 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:50.226025  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 128064us
I20260709 13:03:50.224095  9329 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:50.226962  8656 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:50.204551 14240 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 13:03:50.228834 14091 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:50.229475 14091 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.229919 14091 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:50.231447 14121 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:50.071383 14035 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:50.234967 14161 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.849s	user 0.012s	sys 0.000s
I20260709 13:03:50.210837 14116 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:50.210539 14162 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:50.243090 14242 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.197s
I20260709 13:03:50.204092 14107 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 13:03:50.245785 14176 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.246518 14176 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.252818 14166 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:50.082324 14216 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:50.261996 14216 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:50.266135 14154 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.268106 14154 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:50.273037 14055 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.607s	user 0.005s	sys 0.002s
I20260709 13:03:50.275137 14210 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:50.219686 13948 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.277657 14144 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.278899 14162 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:50.300664  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 73757.5us
I20260709 13:03:50.301470  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:50.277264 14160 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.857s	user 0.006s	sys 0.002s
I20260709 13:03:50.292927  8730 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
W20260709 13:03:50.284480  8650 consensus_meta.cc:287] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.622s	user 0.008s	sys 0.001s
I20260709 13:03:50.292649  8478 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 13, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 13 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:50.309641  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:50.308729  8650 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:50.310756  8478 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:50.286334 14028 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:50.320102  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:50.307682  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:50.303002  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:50.317354  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.322917  8650 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:50.324229  8578 raft_consensus.cc:2368] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10.
I20260709 13:03:50.323829  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:49.917599  8477 rpcz_store.cc:269] 0709 13:03:10.148848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.169691 (+ 20843us) service_pool.cc:224] Handling call
0709 13:03:49.907026 (+39737335us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:35.106708 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:35.107402 (+   694us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:35.107457 (+    55us) write_op.cc:435] Acquired schema lock
0709 13:03:35.107475 (+    18us) tablet.cc:662] Decoding operations
0709 13:03:35.199231 (+ 91756us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:35.199337 (+   106us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:35.199366 (+    29us) tablet.cc:685] Acquiring locks for 29 operations
0709 13:03:35.658882 (+459516us) tablet.cc:701] Row locks acquired
0709 13:03:35.658944 (+    62us) write_op.cc:260] PREPARE: finished
0709 13:03:35.659298 (+   354us) write_op.cc:270] Start()
0709 13:03:35.659507 (+   209us) write_op.cc:276] Timestamp: P: 1783602215659198 usec, L: 0
0709 13:03:35.659563 (+    56us) op_driver.cc:348] REPLICATION: starting
0709 13:03:35.727607 (+ 68044us) log.cc:844] Serialized 11157 byte log entry
0709 13:03:35.986329 (+258722us) op_driver.cc:464] REPLICATION: finished
0709 13:03:36.170339 (+184010us) spinlock_profiling.cc:243] Waited 4.45 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f1406b0c0ac 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f140d64cef0
0709 13:03:36.594706 (+424367us) write_op.cc:301] APPLY: starting
0709 13:03:36.613580 (+ 18874us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:38.127945 (+1514365us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:38.127993 (+    48us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:41.067852 (+2939859us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:41.068614 (+   762us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=58,key_file_lookups=58,delta_file_lookups=58,mrs_lookups=0
0709 13:03:41.068726 (+   112us) write_op.cc:312] APPLY: finished
0709 13:03:41.192800 (+124074us) log.cc:844] Serialized 252 byte log entry
0709 13:03:41.914909 (+722109us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:41.915442 (+   533us) write_op.cc:454] Released schema lock
0709 13:03:41.915543 (+   101us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:41.916226 (+   683us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:41.916260 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":52836,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":554541,"apply.run_cpu_time_us":88623,"apply.run_wall_time_us":5321463,"cfile_cache_hit":118,"cfile_cache_hit_bytes":125289,"num_ops":29,"prepare.queue_time_us":131953,"prepare.run_cpu_time_us":38313,"prepare.run_wall_time_us":723618,"raft.queue_time_us":100823,"raft.run_cpu_time_us":13379,"raft.run_wall_time_us":398465,"replication_time_us":326505,"spinlock_wait_cycles":10224896,"thread_start_us":932692,"threads_started":5,"wal-append.queue_time_us":799911,"wal-append.run_cpu_time_us":20981,"wal-append.run_wall_time_us":2108876}]]}
I20260709 13:03:50.326923  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:50.332089  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:50.341583  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 304744us
I20260709 13:03:50.341606  8478 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 1 to 10, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 10 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:50.344936  8655 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:50.347182  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.348318  8650 raft_consensus.cc:2368] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:03:50.349450  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.351223  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.353113  8650 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:50.351712  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.022s	sys 0.066s
W20260709 13:03:50.354478  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.023s	sys 0.066s
W20260709 13:03:50.355152  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 686791us
I20260709 13:03:50.356134  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:50.357012  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:50.357969  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:50.358839  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:50.290477 14021 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:03:50.287372 14161 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.332484 14221 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:50.370144  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:50.362767  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:50.313205 14239 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12.
I20260709 13:03:50.305747 14174 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 13:03:50.353960 14206 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:50.294761 14055 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:50.375504  8650 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:03:50.364259  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.377451  8478 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 4 to 11. New cstate: current_term: 11 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:50.375988  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:50.398203  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:50.277674 14059 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.400622  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.364391 14030 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:50.363616 14170 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:50.407434  9327 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:50.369557 14210 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:50.404273  8650 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:50.340854 14160 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 13:03:50.368603 13739 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:50.400866 14166 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:50.400080 14169 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:50.400010 14136 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:50.387065 14200 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:50.421308  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:50.450888  8653 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:50.418433  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:50.452310  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:50.430269  8478 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:50.435105  8577 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:50.420631  8731 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:50.413820 14210 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.455574 14210 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:50.429960 14041 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.406044 14062 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 13:03:50.466765 14218 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:50.469835 13948 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:50.470216 14144 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:50.471510 14156 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.538s	user 0.003s	sys 0.008s
I20260709 13:03:50.471319 13930 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:50.472272 14156 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:50.449508 14166 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:50.402864 14201 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:03:50.429718 13739 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:50.492607  9328 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:50.411813 14221 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 13:03:50.505661  8478 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 3 to 13, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 13 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:50.496129 13639 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:50.496141 14144 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.479624 14166 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:50.437260 14121 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 13:03:50.489997 14049 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.479553 14062 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:50.589951  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 369951us
I20260709 13:03:50.578354  8653 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
W20260709 13:03:50.578377  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 358829us
W20260709 13:03:50.553525  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 335381us
W20260709 13:03:50.585202  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 196274us
I20260709 13:03:50.596203  8653 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:50.596647  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:50.603186  8657 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:50.602368  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:50.619128  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:50.620129  8725 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
W20260709 13:03:50.588460 14178 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.749s	user 0.012s	sys 0.000s
I20260709 13:03:50.628831  8658 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:50.630052  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:50.525281 14248 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:50.578140 13930 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:50.630733  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:50.625581 14247 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.262s	user 0.001s	sys 0.000s
I20260709 13:03:50.624629 14161 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:50.618571 14156 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:50.508927 14258 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.130s
I20260709 13:03:50.646076  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:50.646409  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:50.626180 14223 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:50.637306  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:50.678191  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 297183us
I20260709 13:03:50.631089 14221 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:50.678423  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:50.682132  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:50.607249 14049 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.690223 14049 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:50.607458 14062 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.607477 14144 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:50.674010 14099 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.923s	user 0.013s	sys 0.000s
I20260709 13:03:50.698274  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:50.677284 14177 consensus_meta.cc:287] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.868s	user 0.014s	sys 0.001s
I20260709 13:03:50.647274 14156 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:50.697065 14162 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:50.670634 14055 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:50.681124 13930 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:50.681111 14250 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:50.704456 14099 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:50.730376  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 152662us
I20260709 13:03:50.716595 14144 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:50.694391 14257 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:50.712359 14253 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:50.722914 14162 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:50.676791 14021 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:03:50.724274  9322 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1247) took 1432 ms (client timeout 1865 ms). Trace:
W20260709 13:03:50.739883  9322 rpcz_store.cc:269] 0709 13:03:49.291916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.292033 (+   117us) service_pool.cc:224] Handling call
0709 13:03:50.723997 (+1431964us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54006900d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:50.724195 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2979609344}
I20260709 13:03:50.692112 14223 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:50.725282  8654 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 8.
I20260709 13:03:50.726656 14176 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:50.728600 14062 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:50.716492  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2737) took 1093 ms. Trace:
I20260709 13:03:50.746698  8657 rpcz_store.cc:276] 0709 13:03:49.623494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.663727 (+ 40233us) service_pool.cc:224] Handling call
0709 13:03:50.716300 (+1052573us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b540007eed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:50.716426 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1768070400}
I20260709 13:03:50.731830  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:50.748389  8732 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:50.578611 13739 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.703343 14161 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:50.733073  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 338836us
W20260709 13:03:50.738873  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:50.748895  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:50.751513  9325 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:50.734356 14210 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.761133  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:50.761819  9322 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:50.750268  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:50.735513 13930 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:50.770716  8659 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 13:03:50.749893 14161 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.775418 14161 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:50.776741  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2587) took 1331 ms. Trace:
I20260709 13:03:50.744791 14055 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:50.777297  8731 rpcz_store.cc:276] 0709 13:03:49.445546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.445751 (+   205us) service_pool.cc:224] Handling call
0709 13:03:49.446119 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.454382 (+1008263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.776447 (+322065us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b54001c9154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:50.776666 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2240544128}
I20260709 13:03:50.782136 14253 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:50.782891  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2750) took 1010 ms. Trace:
I20260709 13:03:50.784621 14021 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:50.785178 14021 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:50.785703 14021 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:50.783398  8654 rpcz_store.cc:276] 0709 13:03:49.772867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.061369 (+288502us) service_pool.cc:224] Handling call
0709 13:03:50.782671 (+721302us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54001d1ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:50.782809 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1524250368}
I20260709 13:03:50.770165 13930 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.746981 14177 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.794694  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:50.902984 14178 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:50.903545 14178 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.903968 14178 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:50.935429  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:50.962590  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:50.977219  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 306690us
W20260709 13:03:50.985566  8698 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.635s (SENT) [suppressed 21 similar messages]
I20260709 13:03:50.817473 14176 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:50.788671 14250 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:50.813294 14218 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:50.914693 14201 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:03:51.008603  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.028s	sys 0.031s
W20260709 13:03:51.014387  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.028s	sys 0.031s
W20260709 13:03:51.014798  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 659152us
W20260709 13:03:51.018569  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:51.008463 14028 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.000s
W20260709 13:03:51.021561  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 226940us
I20260709 13:03:51.029508  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2723) took 1534 ms. Trace:
I20260709 13:03:51.030543  8658 rpcz_store.cc:276] 0709 13:03:49.494900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.721187 (+226287us) service_pool.cc:224] Handling call
0709 13:03:51.029267 (+1308080us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b540007dad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:51.029434 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1682640768}
I20260709 13:03:51.033645  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:50.998067 14176 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:50.957283 14177 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:50.789275 14156 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.041811  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:51.046582 14099 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:51.024020 14201 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:51.052956 14154 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.784s	user 0.009s	sys 0.004s
I20260709 13:03:51.038841 14028 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:51.058158  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:51.057128 14206 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.650s	user 0.011s	sys 0.000s
I20260709 13:03:50.966065 14059 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.039291 14176 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:51.097126  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 74863us
W20260709 13:03:51.021063  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 479182us
I20260709 13:03:51.019105 14218 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:51.121510  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:51.058943  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 380613us
W20260709 13:03:51.093307 14256 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.423s	user 0.000s	sys 0.001s
I20260709 13:03:51.145681  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:51.149081  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:51.154685  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:51.096405 14200 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.622s	user 0.009s	sys 0.003s
I20260709 13:03:51.007143 14223 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:51.159139  8652 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.010s	sys 0.008s
W20260709 13:03:51.161952  8652 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.010s	sys 0.008s
I20260709 13:03:51.144167 14059 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.141749 14272 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
W20260709 13:03:51.057948 14241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.000s	sys 0.013s
I20260709 13:03:51.175276 14218 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.175494 14210 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:51.176637 14200 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:51.176924 14210 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:51.179056  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1029) took 1283 ms. Trace:
I20260709 13:03:51.179692  8657 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:51.180680  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2749) took 1443 ms (client timeout 1509 ms). Trace:
W20260709 13:03:51.180879  8655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3038) took 1931 ms (client timeout 1635 ms). Trace:
W20260709 13:03:51.188221  8549 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.572s (SENT)
W20260709 13:03:51.188324  8655 rpcz_store.cc:269] 0709 13:03:49.248988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.250016 (+  1028us) service_pool.cc:224] Handling call
0709 13:03:51.180705 (+1930689us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540026b954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:51.180816 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2331587328}
W20260709 13:03:51.188265  8659 rpcz_store.cc:269] 0709 13:03:49.772428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.887705 (+115277us) service_pool.cc:224] Handling call
0709 13:03:51.180474 (+1292769us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b5400380254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:51.180621 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1706885376}
I20260709 13:03:51.186283  8733 rpcz_store.cc:276] 0709 13:03:49.895209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.897556 (+  2347us) service_pool.cc:224] Handling call
0709 13:03:49.943380 (+ 45824us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:49.943838 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:49.944138 (+   300us) log.cc:844] Serialized 14 byte log entry
0709 13:03:50.329805 (+385667us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:50.330254 (+   449us) log.cc:844] Serialized 26 byte log entry
0709 13:03:50.330368 (+   114us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:50.351844 (+ 21476us) log.cc:844] Serialized 14 byte log entry
0709 13:03:50.353770 (+  1926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.358313 (+  4543us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.178801 (+820488us) raft_consensus.cc:1685] finished
0709 13:03:51.178828 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.178993 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":21986,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":2237,"raft.queue_time_us":255980,"raft.run_cpu_time_us":417,"raft.run_wall_time_us":29398,"thread_start_us":385275,"threads_started":1,"wal-append.queue_time_us":645496}
I20260709 13:03:51.099198 14206 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:51.185437  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 913) took 1011 ms. Trace:
W20260709 13:03:51.190815 14217 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.496s	user 0.013s	sys 0.011s
I20260709 13:03:51.194334  8652 rpcz_store.cc:276] 0709 13:03:50.174239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.186439 (+ 12200us) service_pool.cc:224] Handling call
0709 13:03:50.186938 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.187220 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:50.410352 (+223132us) log.cc:844] Serialized 14 byte log entry
0709 13:03:51.185084 (+774732us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:51.185142 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:51.185184 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.185250 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.185358 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20870,"prepare.queue_time_us":398656,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":5906,"thread_start_us":975988,"threads_started":2}
I20260709 13:03:51.197865  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:51.199218  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:51.202067 14284 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.202683 14284 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.203665 14035 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:51.207787  8658 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:03:50.529419 14091 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:51.213716  8730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.007s	sys 0.015s
W20260709 13:03:51.246177  8730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.008s	sys 0.028s
I20260709 13:03:51.247787  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 914) took 1031 ms. Trace:
I20260709 13:03:51.248409  8650 rpcz_store.cc:276] 0709 13:03:50.216367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.431490 (+215123us) service_pool.cc:224] Handling call
0709 13:03:50.431961 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:50.432384 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:50.432428 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:50.433065 (+   637us) log.cc:844] Serialized 26 byte log entry
0709 13:03:50.718394 (+285329us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b180010c660. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:50.987227 (+268833us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:50.987280 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.987499 (+   219us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.209746 (+222247us) raft_consensus.cc:1685] finished
0709 13:03:51.209774 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.247697 (+ 37923us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19225,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":19184,"spinlock_wait_cycles":30098688,"thread_start_us":268584,"threads_started":1,"wal-append.queue_time_us":706617}
W20260709 13:03:51.195298 14217 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.522s	user 0.013s	sys 0.011s
I20260709 13:03:51.060752 14273 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.155s
I20260709 13:03:51.196089 14014 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:50.783460 14253 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.188901 14244 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:51.189815 14036 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:51.052700 14250 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.193759 14210 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.210253 14041 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:51.205821 14026 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:51.287921 14162 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.014s
I20260709 13:03:51.285254 14099 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:51.357061 14284 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:51.293342 14036 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:51.450362 14036 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.451130 14144 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:03:51.416086 14240 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.047s	user 0.008s	sys 0.008s
W20260709 13:03:51.460320 14240 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.095s	user 0.009s	sys 0.009s
I20260709 13:03:51.317683 14099 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:51.429370  8578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 2964) took 2270 ms (client timeout 2130 ms). Trace:
I20260709 13:03:51.338811 14177 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:51.399315  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 413563us
W20260709 13:03:51.442281  8656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2731) took 1886 ms (client timeout 1951 ms). Trace:
I20260709 13:03:51.448117 13948 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:51.321604 14178 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.288620 14014 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.446323  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2412) took 1564 ms. Trace:
I20260709 13:03:51.452253 14036 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:51.462468  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 710741us
W20260709 13:03:51.464556  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 439995us
I20260709 13:03:51.474766 14091 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:51.477665 14244 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:51.477795 14257 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:51.479445 14160 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:51.479943 14246 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.050s	user 0.004s	sys 0.004s
I20260709 13:03:51.480656 14272 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.482788 14206 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:51.431792 14297 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 13:03:51.492703 14099 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:51.492825 14244 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.511668 14244 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.511894 14014 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:51.495586 14257 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.497103 14160 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:51.461583  8578 rpcz_store.cc:269] 0709 13:03:49.158770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.188423 (+ 29653us) service_pool.cc:224] Handling call
0709 13:03:51.428555 (+2240132us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540002dd54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:51.428992 (+   437us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2403561856}
I20260709 13:03:51.484469 13948 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:51.463317  8656 rpcz_store.cc:269] 0709 13:03:49.555943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.615483 (+ 59540us) service_pool.cc:224] Handling call
0709 13:03:51.441782 (+1826299us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b540018e7d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:51.442019 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297504640}
I20260709 13:03:51.518075 14304 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:51.519179 14304 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.521586 14160 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:51.522671  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.523993 13948 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.529917 14301 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:51.521605  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:51.571417  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 416645us
I20260709 13:03:51.576151 14014 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:03:51.572492 14288 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.213s	user 0.000s	sys 0.002s
I20260709 13:03:51.579934  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:51.581211  8650 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:51.581694 14301 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.582875 13948 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:51.585615 14272 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:51.588173 14306 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 13:03:51.593839 14206 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.594916  8650 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:51.606608  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:51.607384  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:51.597915 14251 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.608223  9328 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:51.608685 14251 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.608950  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:51.610003  9322 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 9.
I20260709 13:03:51.612212 14251 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:51.486049  8582 rpcz_store.cc:276] 0709 13:03:49.881979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.882532 (+   553us) service_pool.cc:224] Handling call
0709 13:03:49.909597 (+ 27065us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:49.909860 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:49.909893 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:49.910409 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 13:03:50.379938 (+469529us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:50.406444 (+ 26506us) log.cc:844] Serialized 14 byte log entry
0709 13:03:51.444960 (+1038516us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b180043dc20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287
0709 13:03:51.445319 (+   359us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.445537 (+   218us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.445627 (+    90us) raft_consensus.cc:1685] finished
0709 13:03:51.445747 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.446020 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8490,"prepare.queue_time_us":30399,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":4405,"raft.queue_time_us":223062,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":14288,"spinlock_wait_cycles":6066048,"thread_start_us":743015,"threads_started":2,"wal-append.queue_time_us":551910,"wal-append.run_cpu_time_us":2248,"wal-append.run_wall_time_us":1019187}
I20260709 13:03:51.612823  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2415) took 1220 ms. Trace:
I20260709 13:03:51.586498 14178 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:51.616073  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 912) took 3143 ms. Trace:
I20260709 13:03:51.617027  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:51.617549  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 611005us
I20260709 13:03:51.617985  8655 raft_consensus.cc:2376] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13. Re-sending same reply.
I20260709 13:03:51.619776  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:51.616636  8651 rpcz_store.cc:276] 0709 13:03:48.472904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.474159 (+  1255us) service_pool.cc:224] Handling call
0709 13:03:48.510702 (+ 36543us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:49.462763 (+952061us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:49.462809 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:49.472980 (+ 10171us) log.cc:844] Serialized 26 byte log entry
0709 13:03:50.064160 (+591180us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:50.064201 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.064265 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.478702 (+1414437us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:51.615549 (+136847us) raft_consensus.cc:1685] finished
0709 13:03:51.615571 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.615852 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":217110,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":30937,"spinlock_wait_cycles":10353408,"thread_start_us":1499745,"threads_started":2,"wal-append.queue_time_us":877428}
W20260709 13:03:51.626381  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 133177us
I20260709 13:03:51.630259 14304 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.630489  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:51.631475 14206 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.627390 14258 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:51.633769 14308 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:51.633956 14258 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.635182 14308 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:51.635561 14308 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:51.635663  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:51.636801 14258 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.628060 14144 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:51.613281  8575 rpcz_store.cc:276] 0709 13:03:50.392504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.394891 (+  2387us) service_pool.cc:224] Handling call
0709 13:03:50.395262 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:50.642138 (+246876us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:50.642185 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:50.648481 (+  6296us) log.cc:844] Serialized 26 byte log entry
0709 13:03:51.092379 (+443898us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:51.119941 (+ 27562us) log.cc:844] Serialized 14 byte log entry
0709 13:03:51.611632 (+491691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.611797 (+   165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.611874 (+    77us) raft_consensus.cc:1685] finished
0709 13:03:51.611892 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.612637 (+   745us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":314715,"prepare.queue_time_us":49161,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":461484,"raft.run_cpu_time_us":445,"raft.run_wall_time_us":23085,"thread_start_us":707854,"threads_started":3,"wal-append.queue_time_us":619469}
W20260709 13:03:51.627671 14055 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.850s	user 0.019s	sys 0.002s
I20260709 13:03:51.636845 14272 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.585208 14301 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:51.652902  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:51.659443  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:51.669801  8650 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:51.670068  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:51.673708 14144 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:51.676216 14161 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.876s	user 0.019s	sys 0.002s
I20260709 13:03:51.678568 14144 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:51.679344 14161 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:51.686329  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.687490  9325 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:51.692785 14248 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:51.697955  9322 raft_consensus.cc:2410] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 13:03:51.705878 14248 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.697928  9328 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:51.714397 14248 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:51.733731  8549 leader_election.cc:336] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.191s (SENT)
I20260709 13:03:51.734578  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:51.735929 14121 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:51.718915  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 61354.2us
W20260709 13:03:51.741571  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47414.9us
I20260709 13:03:51.746816  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:51.748580  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:51.749773 14201 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:51.751368 14286 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.461s	user 0.000s	sys 0.002s
I20260709 13:03:51.636698 14308 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.665788 14055 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:51.769644 14303 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:51.774338 14308 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:51.775352 14041 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:51.780092  9325 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:51.785151 13739 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:51.785652 14166 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.141s	user 0.013s	sys 0.010s
I20260709 13:03:51.786607  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:51.786339 14166 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.787745 14201 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:51.790087  9329 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:51.789933  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.790935  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.792706  8575 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:51.793052 14099 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.766851 14267 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:51.797153  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.798101  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:51.798310  8575 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:51.804052  8732 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 13:03:51.804286 14324 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.076s	user 0.000s	sys 0.004s
I20260709 13:03:51.785842 14318 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.805670 14318 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.808602  8577 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:51.809386  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.809986  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:51.810449  8576 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:51.810936  8577 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:51.811520  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.812045  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:51.812718  8577 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:51.813748  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.814535  8583 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:51.817647  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:51.817899  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.818580  9322 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:51.821897  8578 raft_consensus.cc:2410] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
W20260709 13:03:51.824671  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45403.9us
W20260709 13:03:51.773250 14021 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.987s	user 0.010s	sys 0.010s
I20260709 13:03:51.821033  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.830365  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2409) took 1768 ms. Trace:
W20260709 13:03:51.833847  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.837731 14099 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:51.840938 14296 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.492s	user 0.001s	sys 0.000s
I20260709 13:03:51.841368 14318 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.841575 14021 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.838025  8581 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:51.845741  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:51.823838 14166 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.851315  8583 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:51.842903  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:51.860502 14267 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:51.857832 14303 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:51.860899  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:51.861827 14267 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.862417 14303 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.864045  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.864578 14156 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:51.864774 14303 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:51.866297  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:51.867838  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:51.869328 14223 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:51.869954  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:51.870431  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:51.870637  8731 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:03:51.871531 14107 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 13:03:51.872406  8580 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:51.859848  8576 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:51.873828  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:51.874577  8576 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 13:03:51.864802  8581 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:51.875288  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:51.877344 14021 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:51.881986  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:51.882591 14028 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:51.902666  8575 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:51.903489  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:51.832581  8584 rpcz_store.cc:276] 0709 13:03:49.880782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.881010 (+   228us) service_pool.cc:224] Handling call
0709 13:03:49.881485 (+   475us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:49.885927 (+  4442us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:50.308084 (+422157us) log.cc:844] Serialized 14 byte log entry
0709 13:03:50.646921 (+338837us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:50.673412 (+ 26491us) log.cc:844] Serialized 52 byte log entry
0709 13:03:50.673582 (+   170us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:50.673808 (+   226us) log.cc:844] Serialized 14 byte log entry
0709 13:03:50.771623 (+ 97815us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b48002dffd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:03:51.432559 (+660936us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b1800313ba0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:03:51.435291 (+  2732us) log.cc:844] Serialized 14 byte log entry
0709 13:03:51.640874 (+205583us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.641012 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.641093 (+    81us) raft_consensus.cc:1685] finished
0709 13:03:51.641109 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.649517 (+  8408us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":708715,"prepare.queue_time_us":300842,"prepare.run_cpu_time_us":210,"prepare.run_wall_time_us":212,"raft.queue_time_us":1637346,"raft.run_cpu_time_us":5091,"raft.run_wall_time_us":880149,"spinlock_wait_cycles":136046080,"thread_start_us":801691,"threads_started":3,"wal-append.queue_time_us":432394}
I20260709 13:03:51.908109  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:51.909596  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 67046.7us
I20260709 13:03:51.919179  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:51.920047 14318 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:51.920413 14218 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:51.920682 14318 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:51.921141 14218 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:51.921286 14318 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:51.922358 14218 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:51.907177  8581 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:51.928921  8580 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:51.899477 14028 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:51.937192 14059 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:51.939394 14308 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:51.942670 14318 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:51.998734  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 81418.7us
I20260709 13:03:51.943393  8731 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:03:51.949855 14162 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:51.951704  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.952220  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:51.952818  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:51.955282 14303 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:51.955708 14041 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:51.958118  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:51.964195  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58079.3us
I20260709 13:03:51.968187  8582 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:51.971837 14028 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:51.972553  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:51.974377  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.980916  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:51.983546  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:51.987272  9327 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1249) took 1896 ms (client timeout 1674 ms). Trace:
W20260709 13:03:51.993019 14325 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.130s	user 0.000s	sys 0.003s
I20260709 13:03:51.997207  9328 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:51.999472  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.002540  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.003439  8583 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:52.013913  8576 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:51.997980 14091 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:03:52.009552  9327 rpcz_store.cc:269] 0709 13:03:50.090460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.090691 (+   231us) service_pool.cc:224] Handling call
0709 13:03:51.986840 (+1896149us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b5400386b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:51.987077 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":725463936}
I20260709 13:03:52.007675 14308 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:51.998020 14210 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:52.034809  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.036012  8578 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:52.046427  8698 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:52.018199 14091 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:52.050918  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.028183 14028 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:52.055395  9329 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:52.059520  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.035620 14257 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:52.007400 14162 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.070035 14091 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.071203 14162 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.071528 14091 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:52.035782 14161 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:52.078006  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.078794  8581 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:52.079676  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.080312  8581 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:52.061996 14174 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:52.062454 14049 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:52.100545  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:52.101104  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:52.101676  8732 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:52.102347 14216 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.103012 14216 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.105749  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:52.106637  8727 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:52.108978 14301 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:52.054909 14107 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.109434 14301 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.109887 14301 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:52.063113 14308 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.025723 14041 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:52.063139  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.082056  8658 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:52.083557  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.113643  8575 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 8.
I20260709 13:03:52.084656  8651 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 13:03:52.085357  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.085770  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.086586  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:52.061473  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 109284us
I20260709 13:03:52.087136  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:52.063249 14057 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.089226  8582 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:52.089710  8659 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:52.090637  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.013707 14303 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:52.091921 14176 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.853s	user 0.016s	sys 0.003s
I20260709 13:03:52.090672  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:52.092440  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 120028us
I20260709 13:03:52.098698  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:52.099910 14174 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:52.115248  8580 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:52.119107  8732 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
W20260709 13:03:52.122738  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:52.095734  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48014.7us
I20260709 13:03:52.131351  8656 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:52.131896 14197 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:52.132543  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.132725  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.133698  8579 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:52.134472  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.134958  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:52.136104 14257 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:52.136914 14257 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:52.121376  8657 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:52.124373  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.127157  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.128505  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.128641  8576 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:52.150233 14303 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:52.151391  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.151521 14176 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.152257  8651 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:52.152824  8654 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:52.153870 14176 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.085598 14035 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:52.137311  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:52.141338  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.163007  8580 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:52.137688 14162 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:52.167937  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.169211  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:52.170518  8655 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:52.166189 14162 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.172240 14318 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.109826 14107 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.173816 14161 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:52.175347 14107 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.178112  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.178236 14041 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:52.179471  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:52.179718 14041 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.179250 14035 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.180227 14041 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:52.181687  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44511.3us
I20260709 13:03:52.182124 14174 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:52.153775 14049 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:52.185688  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.206547  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:52.207336  8575 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
W20260709 13:03:52.221246  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 179870us
I20260709 13:03:52.221458  8732 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:52.223664  8583 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:52.212878 14057 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.237224 14253 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.238023 14253 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.238749 14057 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:52.239576 14154 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:52.255734  8655 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:52.256948  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.257767  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.259027  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.259914  8659 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:52.263422  8725 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:52.263784  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.264743  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.265237  8724 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:52.266466  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.268252  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3057) took 1235 ms. Trace:
I20260709 13:03:52.268705  8654 rpcz_store.cc:276] 0709 13:03:51.033195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.033364 (+   169us) service_pool.cc:224] Handling call
0709 13:03:52.268036 (+1234672us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54003223d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:52.268178 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2572041472}
I20260709 13:03:52.241247 14035 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.241550 14315 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:52.241732  8658 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:52.243827 14282 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:52.233610 14216 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.213105 14049 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:52.248363 14026 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:03:52.251096  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.251915  8657 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:52.251792  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.278343  8578 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:52.264717  8581 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:52.207229 14303 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:52.270908 14210 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:52.202572 14161 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 13:03:52.289726  8655 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:52.294181  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:52.297694  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.234107 14257 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:52.301076  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:52.305383  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.310524  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.298209  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:52.289290 14174 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.325074  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.325503  8724 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:52.289186 14162 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.287366 14282 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:52.326880  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 192553us
I20260709 13:03:52.340454  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.268281 14301 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.341847  8732 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:52.327531  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:52.347710  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48712.2us
W20260709 13:03:52.364243  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 63279.3us
I20260709 13:03:52.350030 14091 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:52.328202 14210 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:52.327395 14303 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.385515 14226 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:52.386210 14226 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:52.386546 14303 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:52.383077  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 97445.5us
I20260709 13:03:52.385380  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.327252 14294 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.152s
I20260709 13:03:52.386792 14226 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:52.394008  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:52.394363 14218 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:52.394615  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 437898us
I20260709 13:03:52.399302  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.349375 14174 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.401374  8653 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:03:52.394567  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 74365.9us
I20260709 13:03:52.400620 14218 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.408834  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.410176 14218 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.410573 14315 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:52.414551  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.416389  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:52.417682  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.417550 14123 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 13:03:52.420353 14154 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:52.443046  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:52.439888  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:52.446887 14161 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:52.425964 14049 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:52.451661  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.454013 14154 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:52.454587 14154 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:52.455874 14154 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.457769  9327 raft_consensus.cc:2368] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
I20260709 13:03:52.465692  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:52.468648  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.445253 14315 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:52.468928  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.471110  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:52.473279  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:52.468570 14154 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:52.478389  8724 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:52.477422 14218 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.442194 14197 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 13:03:52.480196 14210 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.481885 14218 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:52.482244 14197 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:52.482677 14197 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:52.483569 14197 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.484256 14218 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:52.477720 14123 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:52.484870 14197 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:52.485155 14123 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.485562 14123 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:52.485791  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.487102  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:52.488523 14334 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 13:03:52.489995  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.490855  8658 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 13:03:52.491290  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:52.491806  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 97964us
I20260709 13:03:52.492444 14174 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.493345  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.494791  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:52.493777  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:52.497511 14166 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:52.497844 14257 raft_consensus.cc:2764] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:03:52.499055  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:52.492713  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:52.461978 13948 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 13:03:52.510401  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.513271  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.471942 14161 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 13:03:52.517702  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.514246  8733 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:52.514295  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.522779  8656 raft_consensus.cc:2410] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 13:03:52.544085  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:52.561049  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.571554  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.524128 13948 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:52.578209  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.582114  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.582935  8656 raft_consensus.cc:2376] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 11. Re-sending same reply.
I20260709 13:03:52.567341 14057 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:52.472733 14282 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:52.604313  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.607165  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.616559  8654 raft_consensus.cc:2410] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:03:52.620345  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:52.623487  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.569386 14301 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.627167 14303 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:52.628556 14303 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:52.635910  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.637619  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.580515 13948 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:52.638103  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:52.638103  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.611354 14057 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.671815 14057 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:52.673758  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.674311 13948 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:52.675529 14340 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:03:52.676367 13948 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:52.663156  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.637538  8658 raft_consensus.cc:2376] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 11. Re-sending same reply.
I20260709 13:03:52.660286  8657 raft_consensus.cc:2376] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 11. Re-sending same reply.
I20260709 13:03:52.629679 14267 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:52.688211  8579 raft_consensus.cc:2376] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:03:52.681164 14041 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.697275  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:52.697331  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 131073us
I20260709 13:03:52.629343 14026 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
W20260709 13:03:52.698840  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43900.6us
I20260709 13:03:52.699631  8658 raft_consensus.cc:2368] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:52.700958  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:52.702056  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.702972  8657 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 13:03:52.702919  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.686002 14303 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.684782 14218 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:52.707046  8579 raft_consensus.cc:2368] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12.
I20260709 13:03:52.714342  8581 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:03:52.719328 14218 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:52.720213  8578 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:52.713794 14282 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:52.727766 14248 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:52.728279 14248 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:52.731812 14294 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 13:03:52.731946  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:52.681861 14226 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:52.738410  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.754841  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.756006  8658 raft_consensus.cc:2410] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:03:52.758073  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.732059 14267 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.734150  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:52.764619  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:52.765374  8733 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:52.765748  8651 raft_consensus.cc:2368] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:52.743988 14041 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:52.770656  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.716840 14028 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.778155  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.735677 14227 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:52.781028  8658 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:52.781325  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.788165  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:52.793649 14294 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:52.795198  8730 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:52.796917  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.798627  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:52.798802  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.799103  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.799834  8656 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:52.799535 14028 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.800154 14162 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:52.800631 14162 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.801059 14162 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:52.777170 14041 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:52.803253 14041 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:52.804184 14123 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.805934  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.806779 14123 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:52.807459  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.808153  8657 raft_consensus.cc:2376] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 11. Re-sending same reply.
I20260709 13:03:52.809278  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.810645  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.811245  8577 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
W20260709 13:03:52.811406  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:52.815169  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3060) took 1776 ms (client timeout 1555 ms). Trace:
I20260709 13:03:52.815500 14226 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:52.816044  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:52.816525 14253 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.805949  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:52.817845  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:52.817974 14267 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.782197 14174 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:52.819124 14174 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:52.819327  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.820647 14028 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.820186 14174 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:52.821484  8584 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:03:52.822497 14091 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:52.822923  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 327927us
I20260709 13:03:52.823140 14091 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.823828  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.789274  8725 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:52.820664  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:52.815729  8653 rpcz_store.cc:269] 0709 13:03:51.038652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.041344 (+  2692us) service_pool.cc:224] Handling call
0709 13:03:52.814945 (+1773601us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540018f6d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:52.815095 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2373275008}
I20260709 13:03:52.805202 14041 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:52.839927  8733 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:52.729370 14248 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.785782 14227 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:52.835873 14226 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:52.851104  8732 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:52.855665  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:52.857802  8727 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:03:52.862154  8580 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:52.864434  8576 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:03:52.871793  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 248911us
I20260709 13:03:52.859395 14091 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:52.874784  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.878510  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.879596  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:52.893265  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.901593  8576 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:52.903123  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.906716  8576 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:52.902952  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:52.914911  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.915720 14197 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:52.916330 13948 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:52.916527  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:52.918205 14268 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:52.926570  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 76017us
I20260709 13:03:52.932663  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:52.922318 14343 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.238s	user 0.001s	sys 0.001s
I20260709 13:03:52.936295  8576 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:52.967505  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.968271  8576 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:52.911422 14028 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 13:03:52.969462  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.970337 14028 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:52.971083  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:52.973950  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:52.974048  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47828.3us
W20260709 13:03:52.982685  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66266.1us
W20260709 13:03:52.974573  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 199520us
I20260709 13:03:52.914403 14227 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:52.943764  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:52.989223  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:52.939468 14347 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.939271 14091 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.974704 14248 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:53.002341  8580 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:52.980664 14253 raft_consensus.cc:2764] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:03:53.008134  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.009179 14123 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:52.993288 14154 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.021713 14154 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.024107 14154 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:52.992022 14226 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:52.971164 14028 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 13:03:53.018307 14057 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.006403 14162 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.027402 14057 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:53.027642 14162 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.028329  8659 raft_consensus.cc:2368] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13.
I20260709 13:03:53.028759 14162 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:53.006513 14091 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.024938 14226 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:53.030241  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.029176 14162 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:53.035490  8576 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:53.039498  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.042560  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.045769  9323 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:53.050509 14227 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:53.046622  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.047438  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.048723 14162 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.006696 14347 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.054951  8576 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:53.057196  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:53.058853 14267 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:53.062992 14226 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.068146  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:53.038645  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:53.056231  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.072755  8576 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:53.073812  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.074447  8576 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
W20260709 13:03:53.075779  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 132087us
I20260709 13:03:53.083793  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.084712  8580 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:53.099241  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:53.107641  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:53.108639  8653 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:53.110008  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.110289  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.110072  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:53.111229  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 312689us
I20260709 13:03:53.113379  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.114431  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:53.115160  9329 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.111780  9323 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:53.118623  9322 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:53.112279  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:53.121049  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 154948us
I20260709 13:03:53.091043 14294 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:53.126646  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 208911us
I20260709 13:03:53.127763  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.135275  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:53.136050  8651 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.118494 14154 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:53.138229  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:53.138399  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.141954  8577 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:53.131362  8578 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 7 to 9, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:53.127271  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.145313  8655 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:53.146903  9322 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:53.149529  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 153785us
I20260709 13:03:53.155974  8577 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:03:53.156239  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 146107us
W20260709 13:03:53.158378  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.130928 14267 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:53.174646  9323 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:53.176293  8654 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:03:53.178941  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:53.154155 14210 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.013s	sys 0.011s
I20260709 13:03:53.187747  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:53.188175  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98064.4us
I20260709 13:03:53.183840  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:53.184384  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.201013  8655 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 13:03:53.207849  8549 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.509s (SENT)
I20260709 13:03:53.220955  8654 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 13:03:53.188231 14210 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.013s	sys 0.011s
W20260709 13:03:53.226444  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 47665.3us
I20260709 13:03:53.188314 14123 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:53.179704 14154 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:53.248292  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 126724us
I20260709 13:03:53.248533  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:53.249173  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:53.253718  8657 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
W20260709 13:03:53.254352  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66725.4us
I20260709 13:03:53.255092  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:53.221822 14282 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:53.234709 14353 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:53.290669  8654 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:53.294766  9327 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:53.255335 14055 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:53.218780 14268 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:03:53.308787 14268 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:53.309571 14268 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:53.310878 14268 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.312423 14268 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:53.299798  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 91995.5us
W20260709 13:03:53.313850  8549 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.538s (SENT)
I20260709 13:03:53.314273  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.302261  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:53.321139  9323 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:03:53.329492  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 140636us
I20260709 13:03:53.296753 14123 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:53.221547 14294 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.316506 14091 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:53.331894 14049 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:53.354265 14364 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 13:03:53.356667 14350 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
W20260709 13:03:53.357702 14351 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 13:03:53.359593  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:53.360908  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:53.361514  8579 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:53.362712  8696 leader_election.cc:400] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12.
I20260709 13:03:53.365077  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.365841  9322 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:53.366259  8733 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:03:53.367548  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.364372  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.369048  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:53.369539 14154 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.370491 14353 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:03:53.372171  8579 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:03:53.346377 14123 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:53.374944 14352 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 13:03:53.374534 14123 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.376482  8651 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:03:53.376892 14362 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:53.377421  8581 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:53.378116 14239 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:53.378497 14239 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:53.379354 14239 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.373271  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:53.381898  8575 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
W20260709 13:03:53.382519  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.382663  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.389590  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:53.350894 14361 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:53.405714 14091 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:53.406280  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:53.342633 14375 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:53.380445 14049 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.446777 14375 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.447256 14375 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.448616 14049 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:53.409915 14360 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 13:03:53.410377 14373 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:53.449553 14373 consensus_queue.cc:1243] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:53.451121 14373 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:53.451570 14373 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:53.452041 14373 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:53.332019 14282 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:53.359923 14359 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:03:53.380489 14239 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.413921  9329 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
W20260709 13:03:53.431298  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62335.4us
I20260709 13:03:53.438791  8579 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:53.441697  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 141404us
W20260709 13:03:53.442762  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 196486us
W20260709 13:03:53.444612  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 195521us
I20260709 13:03:53.439528  9327 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:53.443411 14361 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.454910  8576 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:03:53.455667 14049 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 13:03:53.455710  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66234.5us
W20260709 13:03:53.462044  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 97752.4us
I20260709 13:03:53.476713  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:53.477362  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.478092  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:53.473484  8623 leader_election.cc:400] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
I20260709 13:03:53.473372  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:53.444267 14014 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:53.479508  8623 leader_election.cc:403] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:03:53.489161  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:53.495801  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.498435  8548 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:53.479460 14361 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:53.480530 14041 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:03:53.492146 14174 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.005s	sys 0.016s
I20260709 13:03:53.495618 14353 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:53.501864 14253 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:53.514283 14174 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.005s	sys 0.016s
I20260709 13:03:53.479360 14107 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 13:03:53.529853  8579 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:53.530084 14356 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:53.530567 14356 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:53.531066 14356 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:53.533480 14242 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:53.529922 14107 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:53.533998 14242 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:53.534518 14242 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:53.534376 14107 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:53.538149  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.539500  8696 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:53.538873 14107 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.540434 14041 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:53.541990 14041 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:53.542659 14041 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.544521 14375 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.541251 14253 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:53.545471 14353 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:53.545811 14253 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.546509 14253 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.534454 14379 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:53.534417 14389 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.541033  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:03:53.578045  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 264231us
W20260709 13:03:53.583808  8549 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.539s (SENT)
I20260709 13:03:53.584338  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.550333 14107 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:53.501458 14057 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 13:03:53.546540 14353 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.526947 14380 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.091s
I20260709 13:03:53.578006 14379 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:53.634938  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 93985.6us
I20260709 13:03:53.635830  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:53.640172  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 153142us
W20260709 13:03:53.644392  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 64779.1us
I20260709 13:03:53.587433 14026 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:03:53.645157  8549 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.567s (SENT)
I20260709 13:03:53.656993  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.586519 14386 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:53.590991 14174 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:53.601645 14375 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.653829  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:53.662014 14382 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 13:03:53.662483 14388 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:53.673853  8696 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.665140 14373 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.666510 14315 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:53.648697 14253 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.636519 14379 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:53.677089  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 203753us
I20260709 13:03:53.660382 14107 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:03:53.681378  8549 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.607s (SENT)
I20260709 13:03:53.682072  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.600126 14361 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.685035  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:53.687444  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 169784us
I20260709 13:03:53.668886 14386 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.689770  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:53.691488  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.673195 14388 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.677167 14267 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:53.696861  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.676214 14282 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:53.698582 14026 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:53.740373 14026 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:53.740850 14026 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:53.701557 14315 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:53.710582 14356 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.741756 14373 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.695217 14386 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:53.745035  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 935) took 1660 ms. Trace:
I20260709 13:03:53.697533 14268 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:53.745972 14268 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:53.745524  8652 rpcz_store.cc:276] 0709 13:03:52.084795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.150776 (+ 65981us) service_pool.cc:224] Handling call
0709 13:03:52.151074 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:52.204372 (+ 53298us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:52.524552 (+320180us) log.cc:844] Serialized 14 byte log entry
0709 13:03:52.827858 (+303306us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:52.827915 (+    57us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:52.836050 (+  8135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.744738 (+908688us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b540018fbd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:53.744791 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:53.744945 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":446272,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":845,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":6168,"spinlock_wait_cycles":74148096,"thread_start_us":512287,"threads_started":2,"wal-append.queue_time_us":364506}
I20260709 13:03:53.747195 14268 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.748324 14226 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:53.748889 14226 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.702126 14388 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:53.695745 14349 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:53.751040 14226 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.751845 14253 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:53.753798 14026 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.586731 14389 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.757419 14026 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:53.758349 14248 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.750s	user 0.008s	sys 0.007s
I20260709 13:03:53.759037  9328 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:53.732707 14361 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:53.735736  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54402.4us
W20260709 13:03:53.739392 14227 consensus_meta.cc:287] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.688s	user 0.013s	sys 0.000s
W20260709 13:03:53.761439  8549 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.672s (SENT)
I20260709 13:03:53.761984  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:53.766306 14091 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 13:03:53.743561 14339 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.253s	user 0.003s	sys 0.002s
I20260709 13:03:53.767659 14218 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.730150 14041 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:53.779570  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 83407.8us
I20260709 13:03:53.780380  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:53.780627  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.757797 14389 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:53.782367 14227 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.790283  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.743660 14356 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:53.791109  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 240914us
W20260709 13:03:53.778071  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 279707us
I20260709 13:03:53.798207  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:53.798902  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:03:53.800552  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 110881us
I20260709 13:03:53.770467 14041 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:53.804111  8727 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:53.799023  8728 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:53.804412  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:53.806977  8577 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:53.829854  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:53.829847  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:53.835884 14402 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:53.840689  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:53.840909 14402 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:53.841370  9331 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.839655  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.836061  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:53.850832  8578 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260709 13:03:53.851455  8578 pending_rounds.cc:85] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 11
I20260709 13:03:53.796089 14373 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:53.846050  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:53.806324 14400 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:53.786772 14361 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 13:03:53.800505 14218 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:53.807652 14301 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:53.860843  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:53.862221  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.851794  8578 pending_rounds.cc:107] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 13:03:53.865902  8578 raft_consensus.cc:2889] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:53.852432 13739 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 13:03:53.866734  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:53.867465  8654 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.862986 14380 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:53.867982 14123 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.878862  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:53.878777 14123 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.846282 14381 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:53.839738 14091 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:53.842214 14402 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:53.858899 14239 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 13:03:53.869992  8659 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:53.870467  8652 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:53.800866 14227 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.899533  8652 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: [12.9-14.10]
I20260709 13:03:53.877947  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:53.886373  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.917203  8727 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:53.890854  8654 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
W20260709 13:03:53.916721  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 223105us
I20260709 13:03:53.918727  9323 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:03:53.919164  9323 pending_rounds.cc:85] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 11
I20260709 13:03:53.919418  9323 pending_rounds.cc:107] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:03:53.919660  9323 raft_consensus.cc:2889] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:53.893204  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:53.886308  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.885694  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:53.870734  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:53.925593  8653 raft_consensus.cc:2376] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13. Re-sending same reply.
I20260709 13:03:53.901021  9331 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:53.881345 14123 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.888151 14373 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:53.872551 14377 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:53.883853 14055 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:53.885629 14375 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:53.948390  8549 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.700s (SENT)
I20260709 13:03:53.942750  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:53.882896 14402 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:03:53.949537  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63382.6us
I20260709 13:03:53.950294  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.891033 14239 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:53.886059 14355 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:53.945067  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 81590.6us
I20260709 13:03:53.869004 14397 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:53.947598  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:53.881973 14381 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:53.956517  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:53.964721  8727 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:53.957321  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:53.942489 14377 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:53.897578 14301 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:53.885820 14356 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:53.871945 14242 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.960824  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:53.939283 14361 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:53.893029 14026 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 13:03:53.896395 14099 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 13:03:53.891390 14041 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260709 13:03:53.966898  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:53.992460  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44123.4us
W20260709 13:03:53.993700  8549 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.742s (SENT)
I20260709 13:03:53.886194 14218 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:53.949747 14174 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14.
I20260709 13:03:53.882380 14091 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:53.978928  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:54.000972  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53461us
I20260709 13:03:53.971778 14355 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:53.966236 14197 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.979782 14361 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.960249 14380 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:53.896198 14305 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 13:03:54.025046 14197 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:54.028578  8549 leader_election.cc:336] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.743s (SENT)
I20260709 13:03:53.958909 14239 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:53.967692 14377 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:53.967228 14381 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.981851  8728 raft_consensus.cc:1217] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Deduplicated request from leader. Original: 7.8->[10.9-15.11]   Dedup: 10.9->[13.10-15.11]
I20260709 13:03:53.970381 14356 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:54.004747 14166 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:54.005579 14057 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:03:54.007648 14268 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:54.008226  8578 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:03:54.009347  8580 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:54.025107 14361 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:53.972749 14210 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 13:03:53.962291 14400 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:03:54.062407 14355 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.064246 14347 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.937134 14253 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:03:54.014467 14091 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:53.947999 14055 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:54.029469  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 85971.3us
I20260709 13:03:54.045814 14356 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.044248  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.012974 14386 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.054208 14349 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:54.031127 14239 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:54.062674 14388 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.029548 14380 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:54.021998  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.031438  8728 pending_rounds.cc:85] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 9
I20260709 13:03:54.010322  8577 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:54.086565  8728 pending_rounds.cc:107] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 13:03:54.086900  8728 raft_consensus.cc:2889] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:54.075241 14305 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.058660 14362 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.031351 14267 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 13:03:54.074018 14268 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:54.031222 14402 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.089042 14227 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:54.072662 14014 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
W20260709 13:03:54.168802  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.168951  8583 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 13:03:54.084722 14422 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:54.085201 14257 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:54.087960 14380 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.045622 14373 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.195946  8727 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:54.198297  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2986) took 1388 ms. Trace:
I20260709 13:03:54.202759  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.082886 14166 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 13:03:54.205862  8725 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:54.204633  8583 rpcz_store.cc:276] 0709 13:03:52.809677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.813527 (+  3850us) service_pool.cc:224] Handling call
0709 13:03:54.131477 (+1317950us) spinlock_profiling.cc:243] Waited 7.17 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:54.168691 (+ 37214us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:54.198022 (+ 29331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:54.198241 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16517760}
I20260709 13:03:54.194602 14305 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.223392 14305 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:54.225422 14417 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.140s	user 0.002s	sys 0.002s
I20260709 13:03:54.227205 14197 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:54.232087 14379 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.535s	user 0.016s	sys 0.006s
I20260709 13:03:54.225466 14242 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.234092 14379 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:54.236034  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 207479us
W20260709 13:03:54.237936  8549 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.756s (SENT)
I20260709 13:03:54.238541  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.198899 14395 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:54.270778  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1054) took 1471 ms. Trace:
I20260709 13:03:54.272872  8724 rpcz_store.cc:276] 0709 13:03:52.768936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.769259 (+   323us) service_pool.cc:224] Handling call
0709 13:03:52.795048 (+ 25789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:52.838030 (+ 42982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:52.838075 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:52.838127 (+    52us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:52.838171 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:54.240287 (+1402116us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b5400229a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:54.240350 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:54.240537 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75876608}
W20260709 13:03:54.275947  8549 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.775s (SENT)
I20260709 13:03:54.272284  8731 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:54.276665  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.070333 14301 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.286621  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 1018) took 1176 ms. Trace:
I20260709 13:03:54.204758 14422 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:54.287209  9328 rpcz_store.cc:276] 0709 13:03:53.110094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.110236 (+   142us) service_pool.cc:224] Handling call
0709 13:03:54.286404 (+1176168us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b54005dacd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:54.286546 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1440601856}
I20260709 13:03:54.355096  8584 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:54.194819 14362 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.288982 14197 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:54.374702  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 98811.7us
I20260709 13:03:54.381003  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:54.381040  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 181472us
I20260709 13:03:54.383404  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:54.390754  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.391470  8578 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.404049  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.406049  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:54.407193  8583 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:54.367475 14315 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:54.443080  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:54.372653 14347 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:54.444852  8584 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.385886 14402 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.447657  8577 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:54.384902 13948 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:54.452028 13948 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.463757 14055 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.392829 14404 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.465570 14404 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.467864 14404 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.450520 14315 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.471917 14315 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:54.375735 14268 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.477946 14386 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:54.464916 14388 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.465026  8726 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:54.480432 14423 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 13:03:54.480684 14055 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.481750 13948 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.482537 14239 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.451032  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.487499  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.496104  8578 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:54.498715 14218 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.498852 14239 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.445302 14419 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.499748 14419 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.506901  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.507531  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.507671  8727 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:54.508204  8724 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:54.508749  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:54.509347  8733 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:54.510514  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.497885  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.512593  8578 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:54.515329  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.517711  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.523507  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.526520  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.527334  8577 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:54.530651  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:54.552265  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.553550  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:54.559064  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:54.558123  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:54.559100  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.561781  8583 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:54.591804  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 2991) took 1780 ms. Trace:
I20260709 13:03:54.592461  8582 rpcz_store.cc:276] 0709 13:03:52.811488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.813526 (+  2038us) service_pool.cc:224] Handling call
0709 13:03:52.867954 (+ 54428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:52.868256 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:52.868287 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:52.868843 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 13:03:53.098632 (+229789us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:53.098700 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.098788 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:53.578701 (+479913us) raft_consensus.cc:1685] finished
0709 13:03:53.578727 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:54.564209 (+985482us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b5400168ad0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:54.591714 (+ 27505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122675200,"thread_start_us":214860,"threads_started":1,"wal-append.queue_time_us":310160,"wal-append.run_wall_time_us":1409248}
W20260709 13:03:54.603752  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.536978 14347 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:54.607534  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.556169 14055 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:03:54.612470  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 94908.1us
I20260709 13:03:54.616247  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:54.565887 14315 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:54.618347  8725 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:03:54.624238  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:54.591985 14239 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.618300  8578 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:54.604269 14404 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:54.635155  8733 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:54.636473  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:54.638576  8583 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:54.603583 14197 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:54.604980 14431 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:54.631078  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:54.660092  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:54.627254 14315 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:54.661924  8578 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.665069  8582 raft_consensus.cc:2410] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:03:54.662850  9329 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:54.641186 14404 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:54.645560 14049 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.648545 14197 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.678107 14404 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:54.666666 14055 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:54.668820 14315 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:54.669005 14379 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:54.648924 14347 raft_consensus.cc:2764] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 13:03:54.683995 14431 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:54.648474 14257 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:54.696875  8577 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:54.699396 14162 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.692353  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:54.736352  8583 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:54.705204  8578 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:54.642743 14239 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.707839  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.738580  8582 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: [12.9-14.10]
W20260709 13:03:54.741932  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49673.8us
I20260709 13:03:54.743312  8577 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
W20260709 13:03:54.751187  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 127066us
I20260709 13:03:54.752091  8549 leader_election.cc:400] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:54.752461  8549 leader_election.cc:403] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
W20260709 13:03:54.755868  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 58891.5us
I20260709 13:03:54.748368  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:54.739101 13948 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:54.707870 14049 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:54.747599 14361 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.662s	user 0.002s	sys 0.009s
I20260709 13:03:54.748521 14355 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:54.810515  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 102850us
I20260709 13:03:54.816012  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:54.750813 14166 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.086s
I20260709 13:03:54.750818 14413 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.096s
I20260709 13:03:54.748167 14294 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
W20260709 13:03:54.852170  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 100187us
I20260709 13:03:54.916369  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.906957  8584 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
W20260709 13:03:54.885128  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 69201.9us
I20260709 13:03:54.921182  8725 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
W20260709 13:03:54.751796 14380 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.538s	user 0.010s	sys 0.001s
I20260709 13:03:54.919375  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:54.917603  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 172356us
I20260709 13:03:54.753110 14349 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:54.871383 14377 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.816s	user 0.010s	sys 0.000s
W20260709 13:03:54.838868 14445 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 13:03:54.858049 14435 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 13:03:54.814375 14257 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:54.841523 14434 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:54.902654 14447 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:54.963542 14377 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.966327 14413 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:03:54.814828 14361 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:54.966678  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50391.1us
I20260709 13:03:55.007128  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:54.944847 14446 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.813763 14442 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:54.814658 14355 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:54.970780 14049 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:54.974644  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:54.986627 14434 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:54.862187 14432 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 13:03:54.904801 14379 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.898375 14107 raft_consensus.cc:2764] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 13:03:54.964427 14349 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:54.966754 14197 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:54.994514 14381 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.948s	user 0.004s	sys 0.008s
I20260709 13:03:54.975479 14437 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.018356  8651 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:54.955866 14315 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:54.976711 14356 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.873s	user 0.010s	sys 0.000s
I20260709 13:03:55.041898 14257 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.042902 14356 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:55.027845  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66535.8us
I20260709 13:03:54.944550 14380 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.018893 14355 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:54.877238 14055 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.026239 14434 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:55.045315 14381 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.059695 14381 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.060539 14411 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:55.030576 14379 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.054277  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:55.048604 14437 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.054821 14380 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.051630 14431 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.067864 14452 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:55.069051  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2621) took 1113 ms. Trace:
I20260709 13:03:55.068760 14452 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.069515 14437 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.069465  8729 rpcz_store.cc:276] 0709 13:03:53.955627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.955822 (+   195us) service_pool.cc:224] Handling call
0709 13:03:53.968278 (+ 12456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:54.019149 (+ 50871us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:54.023070 (+  3921us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:54.030228 (+  7158us) log.cc:844] Serialized 26 byte log entry
0709 13:03:54.388829 (+358601us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:54.388875 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:54.388957 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:55.068844 (+679887us) raft_consensus.cc:1685] finished
0709 13:03:55.068864 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.068987 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7058,"raft.queue_time_us":44835,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":346423,"threads_started":2,"wal-append.queue_time_us":536312}
I20260709 13:03:55.014941 14446 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.071414  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.071949  8729 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.072959 14446 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:55.084841  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1062) took 1780 ms (client timeout 1735 ms). Trace:
W20260709 13:03:55.085515  8575 rpcz_store.cc:269] 0709 13:03:53.303975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.338823 (+ 34848us) service_pool.cc:224] Handling call
0709 13:03:55.084561 (+1745738us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b54000f82d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:55.084721 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31519104}
I20260709 13:03:55.086410 14377 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.088446 14356 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.091096 14404 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.099529  8729 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
W20260709 13:03:55.102804  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 95796.7us
I20260709 13:03:55.102916  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.104028  8549 leader_election.cc:400] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:55.041985 14455 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:55.031781 14442 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.067327  9325 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:55.048452 14239 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:55.057121 14315 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.061096 14411 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:55.104948  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.104282  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:55.118747  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:55.119148  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.071887 14452 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.122293  9330 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:55.111066  9328 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "266236cfcd824409b757505321b36d81"
dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:39404
I20260709 13:03:55.131134  9328 raft_consensus.cc:606] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:55.131544  9328 raft_consensus.cc:614] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:03:55.138108 14443 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.325s	user 0.004s	sys 0.000s
I20260709 13:03:55.146960 14379 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:55.148253 14457 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 13:03:55.148578 14419 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.151093 14411 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.152839 14411 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.158319  8731 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:55.140108  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2623) took 1183 ms. Trace:
W20260709 13:03:55.176107  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 72254.3us
I20260709 13:03:55.117982 14404 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.182283  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:03:55.183073  8726 raft_consensus.cc:606] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:55.183493  8726 raft_consensus.cc:614] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:55.159591  8728 rpcz_store.cc:276] 0709 13:03:53.956351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.960418 (+  4067us) service_pool.cc:224] Handling call
0709 13:03:53.975164 (+ 14746us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:54.173718 (+198554us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:54.382375 (+208657us) log.cc:844] Serialized 14 byte log entry
0709 13:03:54.664914 (+282539us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:54.667942 (+  3028us) log.cc:844] Serialized 52 byte log entry
0709 13:03:54.668134 (+   192us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:54.668198 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:54.668298 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:54.683326 (+ 15028us) spinlock_profiling.cc:243] Waited 49.1 ms on lock 0x7b180040ed60. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:03:55.062425 (+379099us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:55.139865 (+ 77440us) raft_consensus.cc:1685] finished
0709 13:03:55.139888 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.140047 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62310,"prepare.queue_time_us":47709,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":44,"raft.queue_time_us":160111,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":6156,"spinlock_wait_cycles":127945856,"thread_start_us":335659,"threads_started":2,"wal-append.queue_time_us":341520}
I20260709 13:03:55.196070  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:55.198994 13739 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.198459 14456 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:55.192389 14452 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 13:03:55.246454 13739 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.258190 14452 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:55.258579 14452 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:55.259420 14452 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.260057 13739 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.260689 14452 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:55.250217 14361 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.255192  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1024) took 1418 ms. Trace:
I20260709 13:03:55.262064  9323 rpcz_store.cc:276] 0709 13:03:53.836801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.839759 (+  2958us) service_pool.cc:224] Handling call
0709 13:03:53.840087 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:53.929886 (+ 89799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.254875 (+1324989us) spinlock_profiling.cc:243] Waited 78.3 ms on lock 0x7b54007dc3d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:55.255099 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":180048768}
I20260709 13:03:55.256983  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:55.263654 14055 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:55.264024 14055 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:55.264387 14055 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.264983  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:55.265947 14461 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:55.266417 14461 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.266741 14461 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.257313 14282 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 13:03:55.270869  8583 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:55.271493 14242 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:55.271947 14242 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:55.272230 14349 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:55.272789 14242 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.274308 14242 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:55.275871  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.276579  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:55.277871  9323 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:03:55.278370  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:55.252645 14355 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.280002 14362 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.278321  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:55.283057  8728 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: []
W20260709 13:03:55.284307  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.282222  8583 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:55.286551  8728 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:55.286680  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.296962 14375 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:55.297758 14375 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.299079  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.299913 14375 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.300496 14375 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.300801  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.299930  8654 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:55.301751 14421 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.302066  8655 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 8. Current term is 9. Ops: [8.10-8.10]
I20260709 13:03:55.302285 14421 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:55.302868  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 198665us
I20260709 13:03:55.302191  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:55.308557  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:55.309391  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:55.311573  8657 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:55.314929  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:55.315557  9330 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:55.318969  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:55.319595  8656 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:55.320600  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.321842  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.322402  8728 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:55.323558  8655 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:55.281234 14355 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.334825  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:55.344432  9328 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:55.344220  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.345952  9326 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:55.347299  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.348191  8657 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:55.300989 14375 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.310582 14397 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.376094  8548 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:55.321557 14349 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:55.381058  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.318871 14386 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:55.394342  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.394537 14453 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:55.394582 14386 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.396209  8548 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:55.317205 14057 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 13:03:55.396782 14386 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.397226 14057 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:55.397746 14057 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:03:55.398931 14162 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.657s	user 0.008s	sys 0.003s
I20260709 13:03:55.398614 14057 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.399909 14057 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:55.404424 14461 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:55.401217 14433 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.761s	user 0.000s	sys 0.003s
I20260709 13:03:55.315100 13739 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 13:03:55.473690  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:55.342136 14218 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:55.344971 14226 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:03:55.388581 14434 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.323649 14353 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 13:03:55.389503  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.389973 14405 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:55.393005 14421 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.393726  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.393854  8726 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:03:55.499208  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:55.393930 14375 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.318917 14227 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 13:03:55.360745 14404 raft_consensus.cc:2764] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
I20260709 13:03:55.407611  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.378571 14397 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.505165  9325 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:55.506915 14397 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.408823  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3100) took 1543 ms. Trace:
W20260709 13:03:55.408725  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 231668us
I20260709 13:03:55.507532  8651 rpcz_store.cc:276] 0709 13:03:53.865667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.865802 (+   135us) service_pool.cc:224] Handling call
0709 13:03:53.866111 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:55.034233 (+1168122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.408585 (+374352us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001d1fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:55.408759 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2649192704}
I20260709 13:03:55.508601  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.508857  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.510191  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.510490  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.418855 14395 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:55.510825  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:55.511305  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.511319 14395 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.511765 14305 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:55.511955  8651 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:55.512226 14395 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:55.512511 14305 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.421581  8654 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:55.427192  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.427950  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.382747 14049 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 13:03:55.428279  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.432969 14154 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:55.433959 14412 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:55.435335 14422 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.437532 14381 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.385582  9331 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:03:55.380977  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.448132  8656 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:03:55.449056  8728 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:03:55.516007 14305 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:55.532218  8725 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:55.533532  8657 raft_consensus.cc:2368] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:03:55.534499  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:55.535013  8657 raft_consensus.cc:2376] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16. Re-sending same reply.
I20260709 13:03:55.535118  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:55.535843  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.536545  8657 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:55.537894  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:55.529503  8654 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:03:55.542397  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.543198  8728 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.545150  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:55.549327  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.550197  8655 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:03:55.552536  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.558262  8727 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:55.498247 14253 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:55.349992 14282 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.126s
I20260709 13:03:55.476644  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:55.379850 14356 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.491199  9328 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:03:55.382211 14349 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.507058  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:55.420104 14458 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:55.513785  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.513062  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.580317  8651 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:55.519418  9322 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:55.518424  8732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:03:55.520335  8729 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.520666  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.533242  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.537276  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:55.538869 14434 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.386724 14055 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.520612 14154 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:55.520956 14210 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:55.502034 14267 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.518853 14197 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:55.517992 14257 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:55.599073  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3103) took 1703 ms. Trace:
I20260709 13:03:55.518993 14453 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:55.515096 14461 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.529539 14405 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:55.579492 14349 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:55.548708 14381 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.579564 14253 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.579460  9326 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:55.582768  8656 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:55.583047  9331 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:03:55.581923  8732 raft_consensus.cc:606] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.613583  8732 raft_consensus.cc:614] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:55.589311  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:55.615581  8655 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:03:55.616604  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.590904  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.498116 14218 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.591141  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.552807 14049 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:55.594064  8728 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:55.591688 14154 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:55.593535 14162 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:03:55.600840  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 228062us
I20260709 13:03:55.464514  9330 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:55.587301 13948 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:55.586279 14448 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:55.611217  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 90639.1us
I20260709 13:03:55.624197  8729 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:55.574211 14422 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:55.592494 14210 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:55.640378  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:55.641387  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.651549  8656 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:55.641803  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.642802  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:55.643112  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.646721  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.653429  8655 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:55.645185  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.599594  8659 rpcz_store.cc:276] 0709 13:03:53.896115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.896311 (+   196us) service_pool.cc:224] Handling call
0709 13:03:53.896709 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:53.897152 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:53.897187 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:53.897749 (+   562us) log.cc:844] Serialized 26 byte log entry
0709 13:03:53.958516 (+ 60767us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:53.963100 (+  4584us) log.cc:844] Serialized 14 byte log entry
0709 13:03:55.296628 (+1333528us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b180042c0a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:03:55.592745 (+296117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.592829 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:55.592868 (+    39us) raft_consensus.cc:1685] finished
0709 13:03:55.598798 (+  5930us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.598993 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3063,"prepare.queue_time_us":4544,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":6771,"raft.queue_time_us":1350508,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":22868,"spinlock_wait_cycles":2790656,"thread_start_us":355726,"threads_started":2,"wal-append.queue_time_us":117511,"wal-append.run_cpu_time_us":6591,"wal-append.run_wall_time_us":1212067}
I20260709 13:03:55.587409 14356 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.635818 14434 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.659415  8655 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:03:55.663755  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.664384  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:55.604064 14267 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.660406  8657 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:55.670192 14449 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:55.671604  8659 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:55.672348  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:55.672894  8656 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:03:55.660135  9331 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:55.687070  9326 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:55.670806 14449 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:55.688670 14057 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:55.694387  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:55.694813 14154 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:55.695255  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50159.3us
I20260709 13:03:55.696029  8656 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:55.696105  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.702260  8657 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:03:55.704010  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.592530 14458 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.708150  8549 leader_election.cc:400] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13.
I20260709 13:03:55.709318  9329 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:55.654362 14049 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:55.674486 13948 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:55.740252  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:55.741150  8578 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.650281 14162 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:55.750296 14471 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:03:55.753010 14154 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:55.772537 14469 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.290s	user 0.002s	sys 0.000s
I20260709 13:03:55.776714  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:55.777701 14473 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.778249 14473 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.780311 14473 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.653904 14448 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.804895 14218 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:55.706480 14470 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260709 13:03:55.811831  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.668944 14434 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.813335  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:55.664116 14226 raft_consensus.cc:2764] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:03:55.717231  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:55.830227  8657 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:55.832336  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.833719  8657 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:55.818555 14442 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.836731 14267 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:55.706792 14476 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.140s
W20260709 13:03:55.832404  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 55759.8us
I20260709 13:03:55.720260 14388 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:55.843499  9330 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:55.837785 14413 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:55.839880 14239 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:03:55.722095 14405 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.845474  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:55.719712 14257 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:55.843446  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:55.837630 14218 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:55.860613  8578 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:55.847510 14455 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:55.707099 14395 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.874624 14253 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.876159 14448 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:55.878947 14434 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:55.864401 14049 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.868222 14413 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:55.842175 14305 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:55.882331 14356 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.883309 14381 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:55.846941 14449 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:55.845049  8657 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
W20260709 13:03:55.897945  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54572.6us
I20260709 13:03:55.861941 14055 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:55.906597  8659 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:55.959775  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:55.964757  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:55.910266 14453 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:55.908205  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62886.5us
I20260709 13:03:55.906085 14379 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:55.919435  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1076) took 1611 ms. Trace:
I20260709 13:03:55.966463  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:55.898299  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:55.912954 14267 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:55.910000 14473 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.908104 14388 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:55.911845  8578 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:55.974196  8659 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
W20260709 13:03:55.978926  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 80721.2us
I20260709 13:03:55.990164  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:55.912443 14389 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:55.990861  8733 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:55.996258  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:55.965719 14373 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.979537  8579 rpcz_store.cc:276] 0709 13:03:54.307777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.308139 (+   362us) service_pool.cc:224] Handling call
0709 13:03:54.308634 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:54.309138 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:54.309198 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:54.316826 (+  7628us) log.cc:844] Serialized 26 byte log entry
0709 13:03:55.412434 (+1095608us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b180050b540. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9
0709 13:03:55.414464 (+  2030us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b180050b540. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:55.677470 (+263006us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:55.677545 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.677634 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:55.919151 (+241517us) raft_consensus.cc:1685] finished
0709 13:03:55.919179 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.919362 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9228,"raft.queue_time_us":15401,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":16052,"spinlock_wait_cycles":14099072,"thread_start_us":262751,"threads_started":1,"wal-append.queue_time_us":1545679}
I20260709 13:03:55.911839 14218 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:56.012884  8658 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:55.918604 14458 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.013374  8725 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:55.975183 14091 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:56.016937  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.027486  9296 leader_election.cc:400] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:03:56.027875  9296 leader_election.cc:403] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:56.017360  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.016177  8657 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:03:56.027518  8733 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:56.034282  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:56.030231  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.035296  8724 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:56.032068  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.042197  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.043258  8732 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:56.043203  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.047559  8729 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:55.984256 14267 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:56.055898  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.056144  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:56.056578  8578 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:55.987303 14388 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.907599 14257 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:55.970475 14455 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 13:03:55.985791 14477 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:56.064900  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 454916us
I20260709 13:03:56.024791 14197 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:56.028664 14452 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:55.988672 14055 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:03:55.981313 14453 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.014290 14475 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.075409  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.054961 14226 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:56.067562 14388 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:55.987131 14379 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.055677 14091 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:03:56.070045  8622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.663s (SENT) [suppressed 40 similar messages]
I20260709 13:03:56.026336 14381 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:03:56.054814 14377 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.031523 14239 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:03:56.061285 14267 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:56.050611 14059 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:03:56.031682 14347 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 13:03:56.066394 14412 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:56.025025 14389 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.093909  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:56.095666  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.032235 14373 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.002600 14473 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.099790  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.083266 14421 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 13:03:56.077648 14461 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 13:03:56.100754 14218 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.103544 14377 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.087502 14477 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:56.077371 14154 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 13:03:56.150720  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.166245  8579 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:03:56.076478 14294 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 13:03:56.077920 14166 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 13:03:56.146476  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:56.148511  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 474867us
I20260709 13:03:56.053841 14413 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.150068  8578 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:56.170069  8657 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:56.126344  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:56.146899  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.145927  8581 raft_consensus.cc:2368] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:56.126943  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:56.172577  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:56.110911 14239 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:56.111756 14403 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:56.172320  9323 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:56.091073 14475 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.177973  8579 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:56.177841  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.183431  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:56.137327 14162 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.120129 14395 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.109385 14257 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.177249  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:56.117800 14059 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:56.189764  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:56.191383  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:56.119798 14389 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.135061 14379 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:56.193317  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.196698  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:56.198237  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:56.177224 14239 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:56.199149 14282 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:56.202755  8655 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:56.214320  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:56.179533 14349 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.543s	user 0.012s	sys 0.000s
I20260709 13:03:56.212013  8659 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 10: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:56.177204 14381 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:56.179870 14403 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:56.180419 14405 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:56.189615 14413 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:56.203591 14421 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:56.218220  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.188933 14422 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.250882  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:56.251549  8581 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:56.252492 14453 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.254276 14375 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:56.254726 14375 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:56.255132 14375 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:56.255390 14477 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.257985 14239 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.258172 14210 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.259531 14239 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:56.259631 14210 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:56.207926 14439 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:56.242122 14487 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 13:03:56.208477 14059 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.211452 14253 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:56.218117 14349 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.143386 14091 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.206902 14402 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:56.212329 14305 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 13:03:56.313131  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:56.218945  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:56.330214  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:03:56.320240  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 329796us
I20260709 13:03:56.330968  8578 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:56.332296  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:56.335250  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:56.335879  8650 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:56.314545 14253 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.251437 14422 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.313138 14421 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:56.347883  8651 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:03:56.222208 14405 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.332763 14453 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.354347  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:56.356066  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:56.295562 14282 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 13:03:56.353713  8581 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:56.340255 14439 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:56.350963 14253 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:56.387626  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:56.394748  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.339764 14166 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:56.398731  8575 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:56.400542  8650 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:03:56.383198  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.436398  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.429054  8578 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:56.399350  8576 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:56.467684  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.470976  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.472882  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1079) took 1419 ms. Trace:
W20260709 13:03:56.485744  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.388813 14439 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:03:56.533425  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.537732  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:56.538739  8651 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:56.559370  8733 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:56.561710  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 351748us
I20260709 13:03:56.340216 14403 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.489740  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.492702 14379 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:56.492446 14473 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:56.493780  8583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.024s	sys 0.076s
I20260709 13:03:56.497133  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:03:56.574218  8583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.024s	sys 0.076s
I20260709 13:03:56.575014  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.576267  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.500185 14373 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.501807  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 956) took 1185 ms. Trace:
I20260709 13:03:56.388482 14453 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:56.514416 14491 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 13:03:56.516885  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3105) took 2431 ms. Trace:
W20260709 13:03:56.519671 14498 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 13:03:56.581344  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 376285us
I20260709 13:03:56.565762  8575 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
W20260709 13:03:56.585619  9296 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.016s	sys 0.013s
W20260709 13:03:56.586135  9296 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.016s	sys 0.013s
W20260709 13:03:56.586530  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 509208us
W20260709 13:03:56.568498  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 383669us
I20260709 13:03:56.570418  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.571688  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.506511 14059 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:56.482479  8730 rpcz_store.cc:276] 0709 13:03:55.053901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.054859 (+   958us) service_pool.cc:224] Handling call
0709 13:03:55.055201 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.098525 (+ 43324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.098575 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:55.102096 (+  3521us) log.cc:844] Serialized 26 byte log entry
0709 13:03:55.318047 (+215951us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:55.318127 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.434793 (+1116666us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b5400417154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:56.434856 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:56.434902 (+    46us) raft_consensus.cc:1685] finished
0709 13:03:56.434922 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.472796 (+ 37874us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3103,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":3855,"spinlock_wait_cycles":39316864,"thread_start_us":144959,"threads_started":1,"wal-append.queue_time_us":346876}
I20260709 13:03:56.576273  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2869) took 1110 ms. Trace:
I20260709 13:03:56.555605 14386 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.478922 14166 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:56.438093 14402 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:56.482904  8578 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:03:56.585726  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.582249  8653 rpcz_store.cc:276] 0709 13:03:54.085937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.086277 (+   340us) service_pool.cc:224] Handling call
0709 13:03:54.173789 (+ 87512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.816493 (+1642704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.816532 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:55.833792 (+ 17260us) log.cc:844] Serialized 26 byte log entry
0709 13:03:56.148793 (+315001us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:56.148847 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.148943 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:56.516591 (+367648us) raft_consensus.cc:1685] finished
0709 13:03:56.516616 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.516807 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1201564,"raft.queue_time_us":1149871,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":10079,"thread_start_us":810394,"threads_started":2,"wal-append.queue_time_us":555726}
I20260709 13:03:56.604555  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:56.589172  8576 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:56.610833  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.594955  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.612320  8657 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
W20260709 13:03:56.595259  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.554382 14305 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:56.592213  8731 rpcz_store.cc:276] 0709 13:03:55.465720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.498935 (+ 33215us) service_pool.cc:224] Handling call
0709 13:03:55.499225 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:55.652379 (+153154us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:55.668565 (+ 16186us) log.cc:844] Serialized 14 byte log entry
0709 13:03:55.800779 (+132214us) spinlock_profiling.cc:243] Waited 6.46 ms on lock 0x7b1800369280. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:03:55.983271 (+182492us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:55.983346 (+    75us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:55.983408 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.576063 (+592655us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54001e64d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:56.576115 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.576219 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1772,"prepare.queue_time_us":14415,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":4918,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":26185,"spinlock_wait_cycles":366585600,"thread_start_us":180263,"threads_started":1,"wal-append.queue_time_us":496441}
I20260709 13:03:56.467311 14267 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.577797  8655 rpcz_store.cc:276] 0709 13:03:55.316652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.676750 (+360098us) service_pool.cc:224] Handling call
0709 13:03:56.201339 (+524589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:56.214771 (+ 13432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.501505 (+286734us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b5400059bd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:56.501723 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134814464}
I20260709 13:03:56.566397 13948 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 13:03:56.611492  8576 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:56.641585  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:56.648113  8653 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:56.648164  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.644382  8731 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:56.649735 14154 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:56.608976 14386 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.650523 14154 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.652488  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2786) took 1292 ms. Trace:
I20260709 13:03:56.652916  8656 rpcz_store.cc:276] 0709 13:03:55.359772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.697005 (+337233us) service_pool.cc:224] Handling call
0709 13:03:55.697372 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.699257 (+  1885us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.699284 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:55.743682 (+ 44398us) log.cc:844] Serialized 26 byte log entry
0709 13:03:56.061911 (+318229us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:56.087502 (+ 25591us) log.cc:844] Serialized 14 byte log entry
0709 13:03:56.093586 (+  6084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.093692 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:56.652277 (+558585us) raft_consensus.cc:1685] finished
0709 13:03:56.652299 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.652429 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25346,"prepare.run_wall_time_us":1954,"raft.queue_time_us":6860,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":3769,"spinlock_wait_cycles":97391872,"thread_start_us":312057,"threads_started":1,"wal-append.queue_time_us":574635}
I20260709 13:03:56.642315 14402 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:56.657754 14397 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:56.657491 14402 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.657840 14154 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:56.649168  8651 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:03:56.608172 14475 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.611408  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:56.669219  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:56.599924 14473 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 13:03:56.614677 14403 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:56.581331 14379 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.688220  8581 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:56.672434  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.690268  8731 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:56.652539 14497 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 13:03:56.660773  8730 raft_consensus.cc:2368] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:56.645955 14501 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:56.661556 14498 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:03:56.702796 14422 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:56.702620 14510 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 13:03:56.643352 14305 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.705966 14505 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:56.705960 14305 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.658492 14397 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:56.706131  8733 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:56.675724 14475 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:56.692765 14349 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.671416 14373 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 13:03:56.722877  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:56.678181  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66834.6us
I20260709 13:03:56.626776 14421 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:56.726100  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.627701 14439 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.649729 14166 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.735193  8731 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:56.739733  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:56.727869 14353 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.726831 14405 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:56.727974 14412 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.700028 14501 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:56.737390 14373 consensus_queue.cc:1243] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:56.772161  8730 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:03:56.774986 14494 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:03:56.727639 14375 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:56.784811  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 221197us
I20260709 13:03:56.785745  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:56.725329  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:56.788843  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 239269us
I20260709 13:03:56.796624  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:56.751758 14508 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:03:56.803323  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 585607us
I20260709 13:03:56.803787  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:56.804340  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:56.824121  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:56.850175  8731 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
W20260709 13:03:56.850939  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 59369.8us
I20260709 13:03:56.851689  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:56.853104  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 257072us
W20260709 13:03:56.853242  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49582.2us
I20260709 13:03:56.854084  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:56.864097  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:56.854350 14405 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:56.855319 14449 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.021s	sys 0.037s
I20260709 13:03:56.901444  8654 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:03:56.820544 14373 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:56.849272 14475 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:56.770375 14353 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:56.905220  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 100931us
W20260709 13:03:56.890035 14388 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.791s	user 0.006s	sys 0.006s
I20260709 13:03:56.857125 14506 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:56.874416 14505 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:56.776232 14412 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:56.874617 13948 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:56.889115 14162 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:56.907908 14449 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.022s	sys 0.037s
W20260709 13:03:56.917161 14226 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.826s	user 0.009s	sys 0.004s
I20260709 13:03:56.940905  8731 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:56.941550  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:56.964020  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:56.974538  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:56.969205  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1084) took 1257 ms. Trace:
I20260709 13:03:56.975461  8583 rpcz_store.cc:276] 0709 13:03:55.712303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.713563 (+  1260us) service_pool.cc:224] Handling call
0709 13:03:55.714025 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.714514 (+   489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.714559 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:55.715191 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 13:03:56.574600 (+859409us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:56.574641 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.574717 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:56.968914 (+394197us) raft_consensus.cc:1685] finished
0709 13:03:56.968946 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.969125 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1635,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":22599,"thread_start_us":751955,"threads_started":1,"wal-append.queue_time_us":895279}
I20260709 13:03:56.987252  8576 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:56.988960  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:56.924798 14512 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:56.933571 14267 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:56.931663 14453 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:56.961936 13948 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:56.962637 14162 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:56.995788  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 121808us
W20260709 13:03:56.990506 14496 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 13:03:57.005918  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 154276us
I20260709 13:03:56.983012 14226 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:56.927098 14373 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.026176 14503 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.937215 14388 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:57.027750  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.996392 14305 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:56.991981 14512 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.962996 14506 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.021270 14162 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:56.992447 14505 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:57.019996 13948 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:57.032891 14503 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:57.039366 14516 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 13:03:57.040330 14509 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 13:03:57.034180 14512 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:56.995311 14059 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:03:56.994586 14294 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 13:03:57.027654 14508 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:57.047437  8577 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:03:57.053124  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.103859 14508 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.105464 13948 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.107167  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.107839  8724 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:03:57.107892 13948 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.108668 14412 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.110404  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:57.111402  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.112157 14529 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:57.056545 14373 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.112587 14529 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.973176 14253 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.112974 14529 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:57.114401 14375 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.111404 14353 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.121955 14505 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:57.124152  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:57.072781 14210 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.812s	user 0.011s	sys 0.003s
I20260709 13:03:57.072912 14527 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.058871 14528 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.075824 14386 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.076737 14506 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.082720 14388 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.084709 14503 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:57.085754 14413 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.824s	user 0.005s	sys 0.008s
W20260709 13:03:57.090248 14525 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:03:57.114519 14253 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:57.029245 14501 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.127156  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.138973 14512 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.163726  8724 raft_consensus.cc:2410] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 11 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 10.
W20260709 13:03:57.101253 14239 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.841s	user 0.006s	sys 0.003s
W20260709 13:03:57.161592  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 337474us
I20260709 13:03:57.161841  8725 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:57.162117  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.162040  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.162328  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:57.152686  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 130191us
I20260709 13:03:57.165688  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.166031  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.156819 14049 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.168687  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.168757  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.171324  9323 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:03:57.174919  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.177762  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.178630  8575 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:57.177917  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.182344  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:57.192830  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:57.157186 14527 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.193719  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:57.196833  9323 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:57.157809 14528 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.195572  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:57.194533  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:57.208645  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:57.203565  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.209764  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:57.210179  8575 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:57.210573  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:57.196169 14166 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:57.203775  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.211440  8584 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:57.174352 14253 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:03:57.173996 14049 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.231529  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.232697  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:57.233351  9329 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:57.215941  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:57.216992  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:57.203015 14527 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:57.238673  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.240131  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:57.241638  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.197631 14537 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:57.241982 14166 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.244730 14166 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:57.244813 14489 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.811s	user 0.003s	sys 0.001s
I20260709 13:03:57.217780 14373 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:57.217840 14514 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:57.201498 14506 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:57.226850  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:57.247205 14540 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 13:03:57.285122 14501 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.285049  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.285838  8732 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:57.285730 14501 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.286238  8550 leader_election.cc:400] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:57.286841  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:57.287580  8732 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:57.287837 14501 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.200692 14530 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.227876  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:57.227629 14375 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:57.235420  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:57.289952  9331 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:57.214527 14386 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:57.245810  9326 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:57.241871 14531 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:57.253417  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.258256 14057 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.258746 14166 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.262219  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:57.264273  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.270304  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:57.301124  8696 leader_election.cc:400] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:57.301461  8696 leader_election.cc:403] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:57.302191  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.302801  8579 raft_consensus.cc:2376] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13. Re-sending same reply.
I20260709 13:03:57.302932  9331 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:57.303658  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.304288  8579 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:57.306509  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:57.274601 14528 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:57.275936 14529 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.278046 14512 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.269042  9323 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:57.287459 14388 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:57.361719 14197 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:57.361966 14397 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.362497 14388 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:57.362417 14197 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.362973 14388 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:57.363129 14397 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:57.363325 14529 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.363843 14388 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.364646  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:57.288225  8550 leader_election.cc:403] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:03:57.365756  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 79646.5us
I20260709 13:03:57.366021 14519 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:57.365947  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:57.366708 14519 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:57.367165 14519 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:57.367455  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:57.368075  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:57.288699  8654 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:57.228432  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:57.369813  8651 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:03:57.370177 14538 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:57.371060 14449 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:57.371459 14449 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.366292 14091 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:03:57.371861 14449 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:57.250563  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82375us
I20260709 13:03:57.246294  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.295353 14503 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:57.294502  9329 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:57.377694 14503 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.378232  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.379026  9329 raft_consensus.cc:2368] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:57.379858  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.233464 14422 raft_consensus.cc:2764] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:03:57.380275 14503 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.377429  9322 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:57.298516  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.381342  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.305161  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.382104  8726 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:57.308444  8725 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:57.253518 14373 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.311105  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.312974  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:57.382730  8654 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:57.368160  8729 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:57.383978  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71057.7us
I20260709 13:03:57.313535 14356 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:57.313391  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:57.308917 14442 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:57.314875 14536 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:57.315315 14166 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.319897  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:57.289638 14530 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.299712 14531 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.310081 14057 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.322484  8584 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:57.282094 14506 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:57.323474 14528 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13.
I20260709 13:03:57.324941  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:57.397903  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 73021.3us
I20260709 13:03:57.398651  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:57.310424 14375 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:57.409662 14375 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.305943 14501 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.281095 14162 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.327888 14473 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:57.324865 14379 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.333060  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1302) took 1209 ms. Trace:
I20260709 13:03:57.338624  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:57.339731  9294 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:57.341495 14489 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:57.313637  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.440640  8581 raft_consensus.cc:2376] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13. Re-sending same reply.
I20260709 13:03:57.442113  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:57.443148  8655 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:03:57.445077  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 105431us
I20260709 13:03:57.450277  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:57.345369  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:57.346520  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 962) took 1294 ms (client timeout 1591 ms). Trace:
W20260709 13:03:57.453022  8659 rpcz_store.cc:269] 0709 13:03:56.051852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.059624 (+  7772us) service_pool.cc:224] Handling call
0709 13:03:57.346232 (+1286608us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540018ecd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:57.346445 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":292283264}
W20260709 13:03:57.355171 14504 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.647s	user 0.002s	sys 0.000s
I20260709 13:03:57.359066 14439 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:57.365101 14388 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:57.386960 14373 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:57.382709  8579 raft_consensus.cc:2410] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:03:57.385543  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:57.386538  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:57.390115  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:03:57.467253  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 81776.4us
I20260709 13:03:57.467842  9323 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:57.468001  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:57.388775  8548 leader_election.cc:400] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:57.395231 14531 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:57.380930 14422 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:57.435277 14520 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:57.480149  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:57.385543  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:57.407955 14282 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:57.484270  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.412823  8575 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:57.485010  8579 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:03:57.413549  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:57.414680  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:57.430559  9296 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.591s (SENT)
I20260709 13:03:57.486004  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:57.486481  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71903.9us
W20260709 13:03:57.486667  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 56168.6us
I20260709 13:03:57.438290 14473 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:57.391067 14512 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:57.392367 14536 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:57.389825 14442 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:57.386276 14356 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:57.412221 14506 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:57.457367  8732 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:57.437987 14501 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.375084  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:57.469175  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 80599.5us
I20260709 13:03:57.470400  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:57.470865  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 84442.9us
I20260709 13:03:57.479914 14510 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:57.522953  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:57.390462  9326 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:57.474048  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.475006  8656 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:57.395160  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:57.525714  8581 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:57.526755  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.527318  8581 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:57.477684  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:57.477584  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.529203  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:57.529910  9323 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:57.482347 14477 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:57.481760  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 96291.5us
W20260709 13:03:57.536144  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.536370  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:57.435709  9329 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:57.438670  9324 rpcz_store.cc:276] 0709 13:03:56.123160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.123515 (+   355us) service_pool.cc:224] Handling call
0709 13:03:56.123834 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:56.333276 (+209442us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:56.337432 (+  4156us) log.cc:844] Serialized 14 byte log entry
0709 13:03:56.673764 (+336332us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:56.673819 (+    55us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:56.673861 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.332660 (+658799us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54008f0cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:57.332717 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:57.332886 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30173,"prepare.queue_time_us":6518,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":1856,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":2733,"spinlock_wait_cycles":519302912,"thread_start_us":334164,"threads_started":1,"wal-append.queue_time_us":432990}
I20260709 13:03:57.464514 14386 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:57.458660 14529 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:57.554968  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:03:57.495386 14422 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:57.497190  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.496733 14512 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.560343  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:57.476761 14434 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:57.495244 14532 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:57.480778 14253 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:57.471004 14453 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:57.480573 14538 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:57.529976  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:57.585618  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2466) took 1150 ms. Trace:
I20260709 13:03:57.585986 14538 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.576480 14055 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:57.602028 14545 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
W20260709 13:03:57.608743  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48474.7us
I20260709 13:03:57.480607 14520 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:57.541172  8575 raft_consensus.cc:2410] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 9 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 13:03:57.484292 14282 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.551590  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 940) took 2252 ms. Trace:
W20260709 13:03:57.552193  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 74563.1us
I20260709 13:03:57.551884  9323 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:57.556586  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1090) took 1160 ms. Trace:
I20260709 13:03:57.495445 14536 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:57.614456 14520 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:57.630861 14503 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:03:57.630904 14294 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:57.462366 14379 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:57.633744 14379 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.635025  8584 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:57.636340 14379 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.514991 14356 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:57.637903 14500 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:57.638538 14500 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.466029  8729 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:03:57.563565  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:57.531427 14528 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.644646 14528 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.647449 14528 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:57.555672  8726 raft_consensus.cc:606] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
W20260709 13:03:57.588498 14547 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:03:57.589212 14166 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.594170 14422 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.596545 14501 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:57.561108 14477 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:57.575795 14512 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:57.574766 14434 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.572451 14218 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:57.570488 14510 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:57.569048 14386 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:57.463958 14505 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 13:03:57.586154  8576 rpcz_store.cc:276] 0709 13:03:56.435714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.645333 (+209619us) service_pool.cc:224] Handling call
0709 13:03:56.645646 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:56.997554 (+351908us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.585314 (+587760us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54002cbad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:57.585551 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779318528}
I20260709 13:03:57.557787  8581 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:03:57.611857 14355 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:03:57.625866  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 95059.7us
W20260709 13:03:57.626703  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 96826.9us
I20260709 13:03:57.621729  8578 rpcz_store.cc:276] 0709 13:03:56.395880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.615888 (+220008us) service_pool.cc:224] Handling call
0709 13:03:57.206339 (+590451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.206557 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:57.206588 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:57.206632 (+    44us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:57.206666 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.206719 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:57.556392 (+349673us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54000b5c50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:57.556524 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1356972288}
I20260709 13:03:57.520797 14537 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:57.627751 14282 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:57.615715 14055 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:57.649168  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2798) took 1597 ms. Trace:
I20260709 13:03:57.614297  8652 rpcz_store.cc:276] 0709 13:03:55.298946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.303742 (+  4796us) service_pool.cc:224] Handling call
0709 13:03:55.304018 (+   276us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.436932 (+132914us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:55.660721 (+223789us) log.cc:844] Serialized 14 byte log entry
0709 13:03:56.080564 (+419843us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:56.091293 (+ 10729us) log.cc:844] Serialized 26 byte log entry
0709 13:03:56.091482 (+   189us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:56.091571 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.091675 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:57.244204 (+1152529us) spinlock_profiling.cc:243] Waited 2.93 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:03:57.531118 (+286914us) raft_consensus.cc:1685] finished
0709 13:03:57.551267 (+ 20149us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:57.551506 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":141912,"prepare.queue_time_us":369170,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":146911,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":6665,"spinlock_wait_cycles":6731648,"thread_start_us":586772,"threads_started":2,"wal-append.queue_time_us":659854}
I20260709 13:03:57.806428  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:57.659403  9326 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:03:57.576906 14532 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.578701 14377 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.680678  8548 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:57.695245  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.700695  9323 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:57.607611 14531 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.626850 14253 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:57.787309  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.776491 14528 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:57.798645  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:57.798942  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 276022us
I20260709 13:03:57.671311 14458 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:57.862421 14458 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.797734  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:03:57.617277 14453 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:57.813330  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:57.876605  8576 raft_consensus.cc:2376] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15. Re-sending same reply.
I20260709 13:03:57.805311  8650 rpcz_store.cc:276] 0709 13:03:56.051548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.436067 (+384519us) service_pool.cc:224] Handling call
0709 13:03:57.648927 (+1212860us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b540031cc54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:57.649097 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1276191488}
I20260709 13:03:57.686543 14501 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:57.771715 14434 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:57.817129 14555 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:57.818325 14353 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:03:57.841297  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.788516  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:57.577118 14473 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.796701 14355 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:57.812501 14055 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.943547  9327 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:57.883536  8653 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:57.821626 14532 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:57.826013  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 145407us
W20260709 13:03:57.831687  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.985780  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:57.988103  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 199674us
I20260709 13:03:57.852399 14059 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.865384 14458 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:57.880674 14501 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.866554 14162 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:57.925164 14449 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.002s
I20260709 13:03:58.001344  9327 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:58.001387  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 146063us
I20260709 13:03:58.011958  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:58.012403  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:57.968307 14154 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:57.873821 14528 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.011673 14449 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:58.072222 14551 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.238s	user 0.000s	sys 0.000s
I20260709 13:03:58.072751  8583 raft_consensus.cc:2368] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13.
W20260709 13:03:58.070145  8468 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d0a2b9c938d94a4fa258e068ea054d84 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 15 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:03:57.992736 14555 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:58.053093 14548 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:03:57.992281 14353 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.006278 14473 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:58.062948 14544 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.561s	user 0.002s	sys 0.001s
W20260709 13:03:58.073627  8468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 221751us
I20260709 13:03:58.078895  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1093) took 1517 ms. Trace:
I20260709 13:03:58.079838 14555 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.081250 14057 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:03:58.082755 14091 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.003s
I20260709 13:03:58.079303  8582 rpcz_store.cc:276] 0709 13:03:56.561237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.765419 (+204182us) service_pool.cc:224] Handling call
0709 13:03:56.765945 (+   526us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:56.774871 (+  8926us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:56.774913 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:56.775432 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 13:03:57.315776 (+540344us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b180041a760. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:03:57.481686 (+165910us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:57.483638 (+  1952us) log.cc:844] Serialized 14 byte log entry
0709 13:03:57.575627 (+ 91989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.575740 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:58.078586 (+502846us) raft_consensus.cc:1685] finished
0709 13:03:58.078687 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.078824 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8395,"prepare.queue_time_us":1327,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":138634,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":44406,"spinlock_wait_cycles":13502208,"thread_start_us":257260,"threads_started":2,"wal-append.queue_time_us":1067267}
I20260709 13:03:58.082834 14353 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.084725  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:58.085887  8582 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:58.089191 14449 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.095862 14049 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.097797  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2459) took 1744 ms. Trace:
I20260709 13:03:58.098276  8577 rpcz_store.cc:276] 0709 13:03:56.353828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.353960 (+   132us) service_pool.cc:224] Handling call
0709 13:03:58.097575 (+1743615us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b540002e754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:58.097727 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1562807552}
I20260709 13:03:58.099628  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.100924  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.104656  8584 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
W20260709 13:03:58.105967  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1092) took 1538 ms (client timeout 1657 ms). Trace:
W20260709 13:03:58.106501  8733 rpcz_store.cc:269] 0709 13:03:56.567964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.570072 (+  2108us) service_pool.cc:224] Handling call
0709 13:03:58.105708 (+1535636us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b5400416754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:58.105860 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269483904}
I20260709 13:03:58.110452  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3127) took 1192 ms. Trace:
I20260709 13:03:58.111001  8656 rpcz_store.cc:276] 0709 13:03:56.918490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.338270 (+419780us) service_pool.cc:224] Handling call
0709 13:03:58.110188 (+771918us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x7b540007eed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:58.110368 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80422656}
I20260709 13:03:58.114260  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 1306) took 1173 ms. Trace:
I20260709 13:03:58.116217  9325 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:57.982987 14055 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:58.139423  8577 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:58.141566  8577 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:58.145325  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:58.148833  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:58.149683  8731 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:58.152241  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:58.153290  8731 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:58.155573  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:58.158553  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.163233  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:58.014174  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:58.114876  9330 rpcz_store.cc:276] 0709 13:03:56.940408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.941288 (+   880us) service_pool.cc:224] Handling call
0709 13:03:58.113999 (+1172711us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b540037a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:58.114177 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35836160}
I20260709 13:03:58.171828  8731 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:58.175160  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:58.013357  8581 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:58.001080 14239 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:03:58.071455 14549 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 13:03:58.183549  8726 raft_consensus.cc:2410] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 13:03:58.194281  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 188782us
I20260709 13:03:58.208395  8582 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:03:58.007896 14501 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:58.210080  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:58.210505  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:58.206396 14413 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.057s	sys 0.107s
W20260709 13:03:58.212708  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 414974us
I20260709 13:03:58.213827  8582 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:03:58.204555 14562 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 13:03:57.993535 14477 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:58.241590 14413 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.057s	sys 0.108s
I20260709 13:03:58.165657 14218 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:58.187016 14381 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:58.202112 14458 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.000787 14059 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.190260 14091 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:58.211886 14162 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:58.214661 14449 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:58.275796  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 270724us
I20260709 13:03:58.189926 14561 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:58.284963  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 121826us
I20260709 13:03:58.248845 14512 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:57.997545 14510 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.253465 14500 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:58.324877  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 370792us
I20260709 13:03:58.327503  8651 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:58.329818  8659 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
W20260709 13:03:58.333719  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46439.3us
I20260709 13:03:58.011673 14386 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.349716  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1049) took 1118 ms. Trace:
I20260709 13:03:58.328338 14057 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:58.209185 14355 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.327625 14538 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.320606 14505 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.319554 14154 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:03:58.278788 14218 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.357999  9322 rpcz_store.cc:276] 0709 13:03:57.231157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.232290 (+  1133us) service_pool.cc:224] Handling call
0709 13:03:57.377219 (+144929us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.381330 (+  4111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.349458 (+968128us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400519350. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:58.349637 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319302784}
I20260709 13:03:58.283282 14458 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.330718 14166 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 13:03:58.362428 14500 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:58.282972 14381 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.259377 14253 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.370541 14566 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:58.386426 14355 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:58.399899 14558 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.406656 14267 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:58.406445 14558 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.407202 14267 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.408607 14558 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.343611 14559 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.062s
W20260709 13:03:58.410168 14282 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.585s	user 0.009s	sys 0.000s
I20260709 13:03:58.324115 14449 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:58.388181  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.364190 14561 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:58.398238  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62477.9us
I20260709 13:03:58.366720 14057 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:58.322590 14162 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:58.374095  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 144412us
I20260709 13:03:58.364123  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:58.425731  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:58.426826 14473 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:58.427398 14473 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:58.428527 14473 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.411307  8579 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:58.378005 14294 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.433347 14294 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:58.401134 14439 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.930s	user 0.010s	sys 0.003s
I20260709 13:03:58.368023  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:58.419682  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:58.440016  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.387956 14531 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.453409 14218 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:58.458532 14388 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.982s	user 0.011s	sys 0.000s
I20260709 13:03:58.459446 14154 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.386930 14505 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.466677 14505 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.412298 14500 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.469956 14500 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.470716 14573 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.471258 14573 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.472642 14512 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.439035 14057 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:58.477892  8622 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.619s (SENT)
I20260709 13:03:58.478577  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:03:58.480052 14512 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:03:58.483182 14536 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.860s	user 0.009s	sys 0.003s
I20260709 13:03:58.483870 14536 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.409200 14267 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.437358 14561 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.493968  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:58.443743 14506 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.906s	user 0.011s	sys 0.002s
I20260709 13:03:58.422497 14239 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:58.444025 14403 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:58.445575 14528 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.446385 14162 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.451097 14458 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.492012  8731 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:58.443485 14091 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:58.442160  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 74708.4us
I20260709 13:03:58.410610 14282 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.520193  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:58.505997  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 295521us
W20260709 13:03:58.545940  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 218157us
I20260709 13:03:58.547111  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:58.495498 14397 consensus_meta.cc:287] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.126s	user 0.012s	sys 0.000s
W20260709 13:03:58.512785 14356 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.875s	user 0.008s	sys 0.004s
I20260709 13:03:58.493772 14573 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.563728 14091 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:58.564303 14576 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:58.516584 14434 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.626s	user 0.007s	sys 0.003s
I20260709 13:03:58.604223 14434 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.605676 14239 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.607035 14434 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.516949 14505 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:58.581789  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:58.591768 14519 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.224s	user 0.011s	sys 0.003s
I20260709 13:03:58.607957 14505 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:58.608433 14505 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:58.608181 14519 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.610328 14519 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.593798  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3140) took 1215 ms. Trace:
I20260709 13:03:58.610885  8654 rpcz_store.cc:276] 0709 13:03:57.378220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.896248 (+518028us) service_pool.cc:224] Handling call
0709 13:03:57.896708 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:57.986726 (+ 90018us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:57.986759 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:57.987223 (+   464us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.283047 (+295824us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:58.283085 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.283157 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:58.593526 (+310369us) raft_consensus.cc:1685] finished
0709 13:03:58.593554 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.593720 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58723,"raft.queue_time_us":90790,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9175,"thread_start_us":221223,"threads_started":1,"wal-append.queue_time_us":379523}
I20260709 13:03:58.612720  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.594252 14561 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:58.613655  8622 leader_election.cc:336] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.734s (SENT)
I20260709 13:03:58.614202  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:58.614496 14397 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:58.615487 14397 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:58.613909  8654 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 9: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:58.616926  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.617623 14578 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:58.619108  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:58.619917 14162 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:58.620137 14573 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:58.620654 14573 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.621027 14573 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:58.621333 14162 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:58.622406 14397 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:58.622611 14449 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.557199 14403 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:58.624392 14403 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:58.564429 14565 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 13:03:58.622804 14397 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:58.437848 14566 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.128s
I20260709 13:03:58.613952  8730 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:58.603170 14282 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.617996  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.563088 14439 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:58.629621  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:03:58.635339  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:58.630414  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 644590us
I20260709 13:03:58.503700 14506 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.638223  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2880) took 1407 ms. Trace:
I20260709 13:03:58.638743  8728 rpcz_store.cc:276] 0709 13:03:57.230587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.231204 (+   617us) service_pool.cc:224] Handling call
0709 13:03:57.241843 (+ 10639us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:57.439113 (+197270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:57.439146 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:57.467534 (+ 28388us) log.cc:844] Serialized 26 byte log entry
0709 13:03:57.533826 (+ 66292us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:57.533890 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.627904 (+1094014us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400088ed4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:58.627954 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:58.628010 (+    56us) raft_consensus.cc:1685] finished
0709 13:03:58.628023 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.638142 (+ 10119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13,"raft.queue_time_us":2037,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":6750,"spinlock_wait_cycles":452681984,"thread_start_us":31464,"threads_started":1,"wal-append.queue_time_us":485064}
W20260709 13:03:58.622900 14515 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.774s	user 0.003s	sys 0.010s
I20260709 13:03:58.636229 14581 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:58.642354  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 296377us
W20260709 13:03:58.637946 14520 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.007s	user 0.008s	sys 0.004s
W20260709 13:03:58.642290 14554 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.968s	user 0.002s	sys 0.001s
I20260709 13:03:58.580163 14356 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.663345 14091 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.667608  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3043) took 1177 ms. Trace:
I20260709 13:03:58.668088  8582 rpcz_store.cc:276] 0709 13:03:57.490480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.229553 (+739073us) service_pool.cc:224] Handling call
0709 13:03:58.229912 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.249677 (+ 19765us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.249714 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.250263 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.557442 (+307179us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:58.557486 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.557560 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:58.667358 (+109798us) raft_consensus.cc:1685] finished
0709 13:03:58.667391 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.667541 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23003,"raft.queue_time_us":107029,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":14371,"thread_start_us":306775,"threads_started":1,"wal-append.queue_time_us":415246}
I20260709 13:03:58.656917 14520 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.670168  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:58.670740  8582 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:03:58.671773 14520 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.673085 14381 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.673588 14577 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:58.674057  8650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.005s	sys 0.019s
I20260709 13:03:58.674757  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.677762  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:58.628949 14576 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.679059  9331 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:58.685010  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:03:58.685750  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.685963  9328 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:58.634011 14512 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
W20260709 13:03:58.687302  8650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.006s	sys 0.019s
I20260709 13:03:58.654116 14439 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:58.712086  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:58.715114  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3141) took 1336 ms. Trace:
I20260709 13:03:58.717940  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.717388  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2482) took 1414 ms. Trace:
I20260709 13:03:58.717182  8582 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:03:58.723434  8577 rpcz_store.cc:276] 0709 13:03:57.302700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.142776 (+840076us) service_pool.cc:224] Handling call
0709 13:03:58.143153 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.143406 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:58.420844 (+277438us) log.cc:844] Serialized 14 byte log entry
0709 13:03:58.717035 (+296191us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:58.717088 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:58.717130 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.717204 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.717330 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":371873,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":4227,"raft.queue_time_us":27526,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":87,"thread_start_us":478922,"threads_started":2}
I20260709 13:03:58.724936  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:03:58.726810 14055 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.001s
I20260709 13:03:58.727640 14055 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.718217 14347 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:58.728641  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1286) took 1459 ms. Trace:
I20260709 13:03:58.730449 14356 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.730624 14506 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.731109 14356 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:58.731789 14356 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:58.729159  9326 rpcz_store.cc:276] 0709 13:03:57.269737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.822689 (+552952us) service_pool.cc:224] Handling call
0709 13:03:57.862582 (+ 39893us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.184842 (+322260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.184893 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.201812 (+ 16919us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.585676 (+383864us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:58.594440 (+  8764us) log.cc:844] Serialized 14 byte log entry
0709 13:03:58.595798 (+  1358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.595896 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:58.728426 (+132530us) raft_consensus.cc:1685] finished
0709 13:03:58.728444 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.728584 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5976,"prepare.queue_time_us":7225,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":44182,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":46707,"thread_start_us":522167,"threads_started":2,"wal-append.queue_time_us":463590}
I20260709 13:03:58.732674 14356 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.733942 14439 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.748982 14536 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.684343 14458 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.749799 14458 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.735340  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:58.751677  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:58.673841 14397 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.737784  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:58.753046  9330 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:58.676896 14577 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:58.753549 14397 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:58.753762 14577 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:58.754328 14373 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 1.267s	user 0.009s	sys 0.003s
I20260709 13:03:58.754766 14373 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.759383  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3138) took 1381 ms. Trace:
I20260709 13:03:58.688506 14500 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.760205 14500 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.753077  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1312) took 1443 ms. Trace:
I20260709 13:03:58.764526  9323 rpcz_store.cc:276] 0709 13:03:57.309332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.861447 (+552115us) service_pool.cc:224] Handling call
0709 13:03:57.861808 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.128563 (+266755us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.128611 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:58.128673 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:58.128721 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.752817 (+624096us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54003b01d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:58.752876 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.753014 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":612722560}
I20260709 13:03:58.764878  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.766191  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.767372  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.739832 14267 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:58.768213 14267 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.759858  8650 rpcz_store.cc:276] 0709 13:03:57.377658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.878152 (+500494us) service_pool.cc:224] Handling call
0709 13:03:57.878600 (+   448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:57.920613 (+ 42013us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:57.920663 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:57.978771 (+ 58108us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.688258 (+709487us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:58.731810 (+ 43552us) log.cc:844] Serialized 14 byte log entry
0709 13:03:58.732014 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.732089 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:58.759123 (+ 27034us) raft_consensus.cc:1685] finished
0709 13:03:58.759144 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.759302 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":44608,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":45451,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":38146,"thread_start_us":562304,"threads_started":1,"wal-append.queue_time_us":752775}
I20260709 13:03:58.744421  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1051) took 1512 ms. Trace:
I20260709 13:03:58.684623 14561 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:58.769374  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.769640 14561 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.770030  8650 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:58.770879  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.771454  8650 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:03:58.772250  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.769378  9325 rpcz_store.cc:276] 0709 13:03:57.231723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.232385 (+   662us) service_pool.cc:224] Handling call
0709 13:03:57.380731 (+148346us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1f8c 00007f1406af1ef6 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:58.115697 (+734966us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.116744 (+  1047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.744151 (+627407us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b540037a5d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:58.744347 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1708212352}
I20260709 13:03:58.739533 14055 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:58.733126  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:58.797854  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.761745  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.805987  8654 raft_consensus.cc:2376] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 16. Re-sending same reply.
I20260709 13:03:58.767434  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.818130  8650 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:58.822682  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.824144  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.829371  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.831851  8576 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
W20260709 13:03:58.826830  8549 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.631s (SENT)
I20260709 13:03:58.834267  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:58.828116  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.842458  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.844977  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.824937  8582 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:58.819242 14154 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.815230 14568 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:58.814615 14530 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.859807  8655 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:58.862416  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.864838  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.873126  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:58.850412 14373 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.879683  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.881109  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:58.881724  8576 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:58.883176 14057 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.884102  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2480) took 1581 ms. Trace:
I20260709 13:03:58.884601  8584 rpcz_store.cc:276] 0709 13:03:57.302225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.135894 (+833669us) service_pool.cc:224] Handling call
0709 13:03:58.883416 (+747522us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b5800002798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:03:58.883620 (+   204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.883798 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.883825 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:58.883867 (+    42us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:58.883898 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.883948 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.884048 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10883968}
I20260709 13:03:58.883994  9294 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:58.886256  8650 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:58.887123  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.888008  8654 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:58.889150  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.890605  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.891036 14210 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.891842 14569 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:58.892342  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.895812  8654 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
W20260709 13:03:58.885483 14532 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.882s	user 0.002s	sys 0.019s
I20260709 13:03:58.904131  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.906713  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.904868  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.908226  8584 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:03:58.907539  8654 raft_consensus.cc:2368] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:03:58.864315 14154 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.924875  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.910542  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:58.933162  8650 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:58.938480  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.933754  8654 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:03:58.939420  8577 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:58.936558  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:03:58.941591  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 383236us
I20260709 13:03:58.942526  8576 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:03:58.944772  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:03:58.953883  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 406800us
I20260709 13:03:58.954730  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:58.866560 14568 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:58.913425 14561 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.900296 14210 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:58.981642  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:58.985116  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:58.986157  8730 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:58.987462  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1097) took 2304 ms. Trace:
I20260709 13:03:58.993031  8730 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:58.968236  8650 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:58.975608  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:58.994793  8584 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:58.990686  8580 rpcz_store.cc:276] 0709 13:03:56.683157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.901177 (+218020us) service_pool.cc:224] Handling call
0709 13:03:56.901553 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:56.901773 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:58.771133 (+1869360us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b180014faa0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf
0709 13:03:58.771327 (+   194us) log.cc:844] Serialized 14 byte log entry
0709 13:03:58.987113 (+215786us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:58.987170 (+    57us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:58.987210 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.987281 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.987409 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5431,"prepare.queue_time_us":86182,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":5875,"spinlock_wait_cycles":3548928,"thread_start_us":310448,"threads_started":2}
I20260709 13:03:58.996431  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.956875 14355 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:58.954632 14057 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:58.915700 14532 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:58.998636  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43986.8us
W20260709 13:03:59.000895  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 222792us
I20260709 13:03:59.008247  8654 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:03:59.009222  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:59.015735  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.017103  8650 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 9.
I20260709 13:03:59.026888  8577 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:58.980295 14527 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:59.042315  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:59.039238  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.040549  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.043912  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:58.995244 14413 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:58.987306 14530 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:59.075363 14530 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.076637  8733 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:59.081516 14527 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.084318 14527 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.035913 14573 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.086153  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.086860  8732 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:03:59.087668 14573 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.087988  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.088634  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.088989  8732 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:59.091140 14532 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.093060 14154 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.045578  8654 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:58.989979 14578 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:59.043293  8584 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:59.063081 14055 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 13:03:59.063623 14500 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:59.064982 14589 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 13:03:59.065181 14210 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.066200 14422 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:59.066342  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.066629 14568 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.073467 14576 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:58.986621 13948 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 13:03:59.034737 14267 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:03:59.035344 14501 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.762s	user 0.012s	sys 0.000s
I20260709 13:03:59.074633 14413 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:59.100091  8730 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:59.122167  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.002911 14355 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.143406  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.190471  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:59.191234  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.144357  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.193080  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2812) took 1012 ms. Trace:
I20260709 13:03:59.193451  8650 rpcz_store.cc:276] 0709 13:03:58.180217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.038665 (+858448us) service_pool.cc:224] Handling call
0709 13:03:59.193023 (+154358us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:59.162338 14578 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.164368  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:59.196756  8584 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:59.197201  8725 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:59.171610  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.198028  8654 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:03:59.199303  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:59.201050  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:58.987383 14586 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:59.175892  8726 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:59.177729  8577 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:03:59.037838 14599 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:59.034242 14505 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.173220  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.206483  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.207154  8725 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:59.207717  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:03:59.203717  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.208528  8577 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:59.209931  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.210716  8724 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:03:59.211407  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.214128  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:59.191612 14413 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.192461 14355 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.192977 14210 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:59.180058 14422 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.223402  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.224812  8728 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 14 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:03:59.204963 14057 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:59.219568  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.231599  8654 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:59.212868 14527 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.188827 14501 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.233588  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.209981 14599 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.233707  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:59.243937  8650 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:59.247804  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d851286f2cf248549fc81f8296784c08"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:03:59.248499  8728 raft_consensus.cc:606] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.248865  8728 raft_consensus.cc:614] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:59.240218  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3044) took 1530 ms. Trace:
I20260709 13:03:59.246295  8732 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:59.249540  8584 rpcz_store.cc:276] 0709 13:03:57.709851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.223053 (+1513202us) service_pool.cc:224] Handling call
0709 13:03:59.239413 (+ 16360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.239664 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.239698 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.239764 (+    66us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:59.239828 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.239945 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.240133 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:59.230870 14057 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.255789 14581 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:59.259622  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.260670  8578 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:03:59.278750  8584 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:59.279641  8725 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:03:59.246620 14530 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.285990 14510 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:59.245841 14599 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:59.295768  8657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.042s	sys 0.047s
I20260709 13:03:59.270378 14581 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:59.285207 14577 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.531s	user 0.014s	sys 0.000s
I20260709 13:03:59.299464  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.259907 14057 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:59.231693 14210 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.298311  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.292871  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:03:59.244926 14527 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.304167  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.307382  8650 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:03:59.308461  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.309945 13948 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:59.310873  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.310114  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.304721  8584 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: [14.8-15.9]
I20260709 13:03:59.311767  8650 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:03:59.313006  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.310417 13948 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:59.313952 13948 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.314342  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.314965  8650 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:59.316049  8650 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:59.316668  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3144) took 1306 ms. Trace:
I20260709 13:03:59.316973  8650 rpcz_store.cc:276] 0709 13:03:58.009940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.315575 (+1305635us) service_pool.cc:224] Handling call
0709 13:03:59.315842 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.316416 (+   574us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.316620 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:59.310249 14422 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.306289 14386 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:59.318262  8728 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:59.318449 14386 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:59.318722  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.319387  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.319558 14386 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.320152  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.320664 14294 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.318459  8650 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:03:59.321698  8650 pending_rounds.cc:85] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 6
I20260709 13:03:59.322161  8650 pending_rounds.cc:107] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:59.322383  8650 raft_consensus.cc:2889] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:59.323269  8728 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:59.324139  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2813) took 1143 ms. Trace:
I20260709 13:03:59.324509  8650 rpcz_store.cc:276] 0709 13:03:58.180516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.317878 (+1137362us) service_pool.cc:224] Handling call
0709 13:03:59.318159 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.323910 (+  5751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.324093 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:59.326112  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:03:59.326857  8728 raft_consensus.cc:606] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:03:59.331274  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.333036  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.334411  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:03:59.305171 14600 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 13:03:59.353204 14458 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.349737 14573 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.321702 14059 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.359171  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2488) took 1010 ms. Trace:
I20260709 13:03:59.363802  8584 rpcz_store.cc:276] 0709 13:03:58.348544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.345924 (+997380us) service_pool.cc:224] Handling call
0709 13:03:59.358517 (+ 12593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.358748 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.358787 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.358843 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:59.358893 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.358962 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.359103 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:59.349043  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.366369  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.304953 14581 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:59.344388 14530 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:59.367748 14530 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:59.367935 14576 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:59.368558 14576 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:03:59.373790 14197 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.404098  8728 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:59.399083  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:03:59.306900 14577 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.347939 14294 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:59.350670 14461 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:59.427724  8658 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:59.354333 14566 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 13:03:59.354115 14598 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.408773  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:59.362279 14453 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.453608 14538 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.454377 14538 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.456818 14538 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:59.452131  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43492.7us
I20260709 13:03:59.358103 14573 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.459959  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.434267 14532 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:59.480537 14605 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.265s	user 0.001s	sys 0.001s
I20260709 13:03:59.437110 14577 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.479864 14166 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.425977 14055 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 13:03:59.458000 14569 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.480747 14512 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:59.492168 14611 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 13:03:59.520709 14461 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.489040 14578 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.425709 14294 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:03:59.525187  8579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3038) took 2035 ms (client timeout 1505 ms). Trace:
I20260709 13:03:59.539722  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:59.539621  8579 rpcz_store.cc:269] 0709 13:03:57.489372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.497000 (+  7628us) service_pool.cc:224] Handling call
0709 13:03:59.525011 (+2028011us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b54002d78d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.525118 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1740077312}
I20260709 13:03:59.527182  8581 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:03:59.542434  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2483) took 2239 ms. Trace:
I20260709 13:03:59.528460  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3129) took 2609 ms. Trace:
W20260709 13:03:59.545382  9329 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1283) took 2289 ms (client timeout 1787 ms). Trace:
I20260709 13:03:59.545765  8659 rpcz_store.cc:276] 0709 13:03:56.918949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.464476 (+545527us) service_pool.cc:224] Handling call
0709 13:03:57.464829 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.332405 (+867576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.528119 (+1195714us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54000c80d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:59.528385 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1988334464}
I20260709 13:03:59.547236  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.548393  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.533499  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3137) took 2156 ms. Trace:
W20260709 13:03:59.527233  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.549809  8655 rpcz_store.cc:276] 0709 13:03:57.377365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.860532 (+483167us) service_pool.cc:224] Handling call
0709 13:03:57.913820 (+ 53288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.863531 (+949711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.529335 (+665804us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b540016e4d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:59.533348 (+  4013us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2174808960}
I20260709 13:03:59.542928  8581 rpcz_store.cc:276] 0709 13:03:57.302943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.207467 (+904524us) service_pool.cc:224] Handling call
0709 13:03:58.242327 (+ 34860us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:59.526822 (+1284495us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b5400173ca0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:03:59.542114 (+ 15292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.542367 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46566784}
W20260709 13:03:59.545856  9329 rpcz_store.cc:269] 0709 13:03:57.249353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.379592 (+130239us) service_pool.cc:224] Handling call
0709 13:03:59.538835 (+2159243us) spinlock_profiling.cc:243] Waited 55.2 ms on lock 0x7b54008b4054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.538964 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127069952}
I20260709 13:03:59.553814  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:59.555059  8659 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.521795 14512 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:59.565979  8579 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:59.566119  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.567066  8655 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:03:59.561381  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.521325 14166 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.459486 14598 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.481227 14576 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.524996 14569 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.568944  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:59.559439 14610 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
I20260709 13:03:59.575549  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.586027  8732 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:03:59.563964 14512 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:59.569813 14501 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.541917 14461 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.539301 14532 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.592228  8659 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:03:59.600812  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:59.611284  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 221862us
I20260709 13:03:59.585780 14505 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.585340 14397 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:59.571599 13948 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.601558  8655 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 10.
I20260709 13:03:59.616565  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.585319 14538 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:03:59.592592 14609 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 13:03:59.620905  8725 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:59.620932  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.579864 14503 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:59.562974 14599 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.625658  8659 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:03:59.625840  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 964) took 3019 ms. Trace:
I20260709 13:03:59.632440  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.631289 14538 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.635275 14538 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.635577 14581 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.636152 14413 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.638863  8650 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:59.640069 14599 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.640456 14503 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.641042 14057 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.641319  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3128) took 2722 ms. Trace:
I20260709 13:03:59.673861  8651 rpcz_store.cc:276] 0709 13:03:56.918713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.370507 (+451794us) service_pool.cc:224] Handling call
0709 13:03:57.370794 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.327997 (+957203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.641081 (+1313084us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54001d1fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:59.641260 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2199500544}
I20260709 13:03:59.641562  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:03:59.642482  8578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3041) took 2152 ms (client timeout 1771 ms). Trace:
W20260709 13:03:59.675575  8578 rpcz_store.cc:269] 0709 13:03:57.490091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.798401 (+308310us) service_pool.cc:224] Handling call
0709 13:03:59.642279 (+1843878us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540002e254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.642406 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3234504448}
I20260709 13:03:59.643075  8659 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:03:59.677264  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.678277  8659 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:03:59.679281  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.680024  8659 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:03:59.644057 14566 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:59.644933  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.645355 14581 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.637662 14442 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.647281 14059 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:03:59.649712 14055 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:03:59.649292 14530 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.651973  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2646) took 2491 ms. Trace:
W20260709 13:03:59.656701 14607 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.423s	user 0.001s	sys 0.000s
I20260709 13:03:59.657836  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:03:59.665380  8549 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.772s (SENT)
W20260709 13:03:59.699184  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 78302.9us
I20260709 13:03:59.700762  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.631989  8657 rpcz_store.cc:276] 0709 13:03:56.606251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.623701 (+ 17450us) service_pool.cc:224] Handling call
0709 13:03:56.624070 (+   369us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:56.659980 (+ 35910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:56.660013 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:58.754355 (+2094342us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b2c000b1fc8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:03:58.754912 (+   557us) log.cc:844] Serialized 52 byte log entry
0709 13:03:59.301676 (+546764us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:59.583867 (+282191us) log.cc:844] Serialized 14 byte log entry
0709 13:03:59.584123 (+   256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.584274 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.603705 (+ 19431us) raft_consensus.cc:1685] finished
0709 13:03:59.624976 (+ 21271us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.625121 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":327986,"raft.queue_time_us":89323,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":7492,"spinlock_wait_cycles":4384512,"thread_start_us":804191,"threads_started":2,"wal-append.queue_time_us":681453}
I20260709 13:03:59.685534  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.683432 14613 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:59.686874 14413 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.687402 14573 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.628041 14397 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.687646 14598 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.703795 14530 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:59.706300  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.709715  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.710721  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:59.711656  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.712651  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.689136 14532 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.692022 14569 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:59.698299  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53430.8us
I20260709 13:03:59.695474  8727 rpcz_store.cc:276] 0709 13:03:57.160794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.161010 (+   216us) service_pool.cc:224] Handling call
0709 13:03:58.105894 (+944884us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.106162 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:58.413559 (+307397us) log.cc:844] Serialized 14 byte log entry
0709 13:03:58.724040 (+310481us) log.cc:844] Serialized 14 byte log entry
0709 13:03:58.820600 (+ 96560us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:58.820680 (+    80us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:58.820752 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.820853 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.651669 (+830816us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b5400416750. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.651831 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":130005,"prepare.queue_time_us":488714,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":89,"raft.queue_time_us":146252,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":8027,"spinlock_wait_cycles":2145010432,"thread_start_us":581702,"threads_started":2,"wal-append.queue_time_us":458078}
I20260709 13:03:59.713614 14581 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:59.714139 14581 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:03:59.714166  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:03:59.714699 14581 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:59.706926 14619 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:59.707206  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:03:59.717021 14210 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.701855  8651 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:03:59.702950  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.702880  9296 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:59.737946  8651 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:03:59.738564  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 968) took 1058 ms. Trace:
I20260709 13:03:59.738847  8651 rpcz_store.cc:276] 0709 13:03:58.679932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.737412 (+1057480us) service_pool.cc:224] Handling call
0709 13:03:59.737735 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:59.738321 (+   586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.738507 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:59.739063  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.699186 14055 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.681071  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:03:59.740617  8659 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:03:59.717198  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3148) took 1215 ms. Trace:
I20260709 13:03:59.741612  8655 rpcz_store.cc:276] 0709 13:03:58.501314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.716457 (+1215143us) service_pool.cc:224] Handling call
0709 13:03:59.716743 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.716901 (+   158us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.716926 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.716967 (+    41us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:59.716998 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.717046 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.717148 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:59.742353  8575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2471) took 2445 ms (client timeout 1619 ms). Trace:
I20260709 13:03:59.707406  8579 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:59.742803  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:03:59.742767  8575 rpcz_store.cc:269] 0709 13:03:57.296982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.297082 (+   100us) service_pool.cc:224] Handling call
0709 13:03:59.742171 (+2445089us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b54003a4654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.742290 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":67177344}
I20260709 13:03:59.701855  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.717891 14527 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.751845  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.734740 14503 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.753255  8659 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:59.753820 14166 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.754999  8659 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:59.756201  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.719039 14577 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:59.722816  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2659) took 1548 ms (client timeout 1976 ms). Trace:
W20260709 13:03:59.757586  8733 rpcz_store.cc:269] 0709 13:03:58.174442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.174805 (+   363us) service_pool.cc:224] Handling call
0709 13:03:59.722602 (+1547797us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54004727d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.722753 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1991950080}
I20260709 13:03:59.759518  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2814) took 1578 ms. Trace:
I20260709 13:03:59.759953  8650 rpcz_store.cc:276] 0709 13:03:58.208137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.326712 (+1118575us) service_pool.cc:224] Handling call
0709 13:03:59.326986 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.666392 (+339406us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.759201 (+ 92809us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b5400001754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:59.759452 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":716698880}
W20260709 13:03:59.723517  8583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2473) took 2425 ms (client timeout 1734 ms). Trace:
W20260709 13:03:59.761595  8583 rpcz_store.cc:269] 0709 13:03:57.298144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.298256 (+   112us) service_pool.cc:224] Handling call
0709 13:03:59.723318 (+2425062us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b54003a3c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.723453 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588079488}
I20260709 13:03:59.726706  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:03:59.763579  8577 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:59.763924 14569 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:59.764752  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:59.703574 14442 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.691689 14512 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:03:59.767617  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3125) took 2826 ms (client timeout 3084 ms). Trace:
I20260709 13:03:59.768132 14512 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:03:59.768043  8653 rpcz_store.cc:269] 0709 13:03:56.917970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.344991 (+427021us) service_pool.cc:224] Handling call
0709 13:03:59.743947 (+2398956us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.744065 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":990119680}
I20260709 13:03:58.715879  8656 rpcz_store.cc:276] 0709 13:03:57.378491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.111985 (+733494us) service_pool.cc:224] Handling call
0709 13:03:58.112455 (+   470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.129863 (+ 17408us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:58.505300 (+375437us) log.cc:844] Serialized 14 byte log entry
0709 13:03:58.714755 (+209455us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:58.714800 (+    45us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:58.714850 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.714913 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.715059 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":524645,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":26320,"thread_start_us":481239,"threads_started":2}
I20260709 13:03:59.769789  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.770361  8656 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:59.771011  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.771654  8725 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:03:59.771848  8656 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:59.706401 14294 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.735267  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:03:59.773336  8728 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:03:59.708181  8581 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:03:59.773838 14586 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:59.706799  8578 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:59.774441 14619 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:59.775158 14538 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:59.746855  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:59.776744 13948 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.777552  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:59.776355 14055 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.776243  8733 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:03:59.698258 14059 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.763408  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:59.774843 14619 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.707600 14057 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:03:59.781751 14619 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:59.782313  8653 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:03:59.779489  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.783748  8653 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:59.747242  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:03:59.786317  8656 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:59.786888  9327 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:03:59.787820  8577 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:03:59.788378  9331 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:03:59.788517  8549 leader_election.cc:400] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:03:59.788909  8549 leader_election.cc:403] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:03:59.729925 14355 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.791218  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.793025  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.795625  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.796905  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.794625 14501 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:59.801604 14197 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:59.810485 14397 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:03:59.815308 14355 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.816315 14461 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.820158  8728 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:59.822132  8733 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:59.822510  8733 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:03:59.823308  8733 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:03:59.824280  8733 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:59.824406  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.826550  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3145) took 1401 ms. Trace:
I20260709 13:03:59.826953  8657 rpcz_store.cc:276] 0709 13:03:58.425029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.717930 (+1292901us) service_pool.cc:224] Handling call
0709 13:03:59.737373 (+ 19443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.737574 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.737601 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.737640 (+    39us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:59.737669 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.737738 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.826366 (+ 88628us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b5400470750. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.826498 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44056320}
I20260709 13:03:59.827903  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:03:59.828006  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 80867.2us
I20260709 13:03:59.829453  8724 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:59.829742  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:59.836107  8656 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 13:03:59.836661  8656 pending_rounds.cc:85] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 8
I20260709 13:03:59.836936  8656 pending_rounds.cc:107] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 13:03:59.837216  8656 raft_consensus.cc:2889] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:59.837318  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44362.4us
W20260709 13:03:59.843968  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47115.6us
I20260709 13:03:59.849697  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.849953  9328 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:59.845288  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.856649  8698 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.858697  8698 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.795401 13948 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:03:59.860108  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:03:59.860786  9331 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:59.865525  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.865852  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:03:59.850380 14601 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:59.870824 14601 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.871184 14601 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:59.871506  8727 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:03:59.873205 14566 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 13:03:59.875066  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3146) took 1449 ms (client timeout 1704 ms). Trace:
I20260709 13:03:59.875168 14059 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:03:59.849313 14403 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 13:03:59.876591 14403 consensus_queue.cc:1243] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:59.826305 14586 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:59.878392  8725 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:59.880913  8655 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:59.881934  9328 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:03:59.875607  8658 rpcz_store.cc:269] 0709 13:03:58.425363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.767109 (+341746us) service_pool.cc:224] Handling call
0709 13:03:59.874874 (+1107765us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b540025bf54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:03:59.875008 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1512347904}
I20260709 13:03:59.886948  9331 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:03:59.877458 14577 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:59.894320  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:59.899979  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:59.853152 14458 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:59.883052 14530 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:59.880971 14439 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:59.877873 14626 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:59.907325  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2660) took 1709 ms. Trace:
I20260709 13:03:59.890048 14267 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:59.911053  8731 rpcz_store.cc:276] 0709 13:03:58.198194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.199783 (+  1589us) service_pool.cc:224] Handling call
0709 13:03:58.200195 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.512254 (+312059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.907055 (+1394801us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b5400338e54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:03:59.907267 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":669870976}
I20260709 13:03:59.895180 14403 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 13:03:59.912518 14626 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.913250 14626 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:59.913488 14566 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:59.914129 14566 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:59.892072 14577 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:59.915230 14566 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.916443 14239 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.916105 14577 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.916802 14566 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.917151 14577 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:59.918501 14473 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:59.894550 14356 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 13:03:59.854210 14397 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 13:03:59.923774 14397 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:59.924175 14397 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:03:59.924980 14397 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.916965 14239 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:03:59.926246 14397 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:59.933285  8728 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:03:59.934027  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:03:59.935180  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:03:59.937326  9326 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:59.909992  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:03:59.942209  9328 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:03:59.940367  8548 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:03:59.949525  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.951741  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:03:59.953128  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 72607.5us
W20260709 13:03:59.971578  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 255352us
I20260709 13:03:59.909584 14501 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:59.986368  9328 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 13:03:59.985440  9326 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 13:03:59.986508  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.949877 14057 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:59.910845 14439 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:59.989617  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.973711  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.936367 14267 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:03:59.993566 14267 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:59.944796 14059 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:59.992844  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:03:59.998806  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.000298  9328 raft_consensus.cc:2368] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:03:59.982153  8733 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 12. (term mismatch)
I20260709 13:04:00.001057  8733 pending_rounds.cc:85] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 11
I20260709 13:04:00.001296  8733 pending_rounds.cc:107] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 12.12
I20260709 13:04:00.001538  8733 raft_consensus.cc:2889] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:00.011492  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:00.017563 14519 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 13:03:59.991575 14057 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.021874 14057 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.023753 14434 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:00.025434 14569 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.026259 14569 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.027192 14253 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:59.995378 14439 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.033584 14055 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:59.983783 14572 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
W20260709 13:04:00.040082  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66462.8us
I20260709 13:03:59.988567 14530 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:00.029047 14569 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:00.059365  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.102317  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:00.004184 14527 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:59.998440 14059 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.036728 14448 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:00.149431  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 304095us
I20260709 13:04:00.158285  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.162153 14577 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:00.108001 14573 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.150861 14527 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:00.153968 14059 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:00.185292  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.154224 14448 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.174923  9326 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:00.163055 14629 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:00.184712 14527 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:00.174919  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 982) took 1096 ms. Trace:
I20260709 13:04:00.292409  8650 rpcz_store.cc:276] 0709 13:03:59.078202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.760802 (+682600us) service_pool.cc:224] Handling call
0709 13:03:59.761053 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.761268 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:00.036703 (+275435us) log.cc:844] Serialized 14 byte log entry
0709 13:04:00.174422 (+137719us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:00.174517 (+    95us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:00.174589 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.174676 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.174843 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2722,"prepare.queue_time_us":140635,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":2803,"thread_start_us":331680,"threads_started":2}
W20260709 13:04:00.175555  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 131882us
I20260709 13:04:00.420511  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.231305 14448 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:00.455711  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 160252us
W20260709 13:04:00.291661 14581 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.524s	user 0.007s	sys 0.004s
I20260709 13:04:00.487583  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.177390 14573 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.463750 14154 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.502116  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:00.506498  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.507371  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:00.454645 14629 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:00.230855 14530 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:00.517262  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:00.518091 14572 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:00.518857 14536 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:04:00.523694 14619 consensus_meta.cc:287] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.741s	user 0.005s	sys 0.004s
I20260709 13:04:00.501328 14644 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:00.537470 14630 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.374s	user 0.002s	sys 0.000s
W20260709 13:04:00.539880 14601 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.668s	user 0.007s	sys 0.003s
I20260709 13:04:00.479597 14566 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:00.510959 14636 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.220s	user 0.000s	sys 0.000s
I20260709 13:04:00.490298 14581 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.543357 14154 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:00.549435 14626 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.636s	user 0.007s	sys 0.003s
I20260709 13:04:00.558667 14569 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:00.553279 14578 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:04:00.557565  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50268.3us
I20260709 13:04:00.537622 14305 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 13:04:00.581688 14536 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:00.587989 14578 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:00.590126  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.589684 14626 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.595901  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 967) took 1942 ms. Trace:
I20260709 13:04:00.596596 14626 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:00.594332  8651 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:00.576953 14601 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:00.598217  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 969) took 1877 ms. Trace:
I20260709 13:04:00.598577  8651 rpcz_store.cc:276] 0709 13:03:58.720882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.752020 (+1031138us) service_pool.cc:224] Handling call
0709 13:03:59.752240 (+   220us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:00.597155 (+844915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.597997 (+   842us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b540018fbd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:00.598161 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1935906048}
I20260709 13:04:00.595815 14578 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:00.599485  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.574287 14644 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.594808 14536 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:00.602476 14644 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:00.590283  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.581615 14566 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.602984 14536 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:00.603924 14625 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.696s	user 0.000s	sys 0.013s
I20260709 13:04:00.604429 14536 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.599226 14601 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:00.605351 14566 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:00.605684 14631 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.140s	user 0.002s	sys 0.000s
I20260709 13:04:00.596331  8652 rpcz_store.cc:276] 0709 13:03:58.656044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.724125 (+1068081us) service_pool.cc:224] Handling call
0709 13:03:59.724458 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:59.724786 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.724814 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:59.725252 (+   438us) log.cc:844] Serialized 26 byte log entry
0709 13:03:59.949875 (+224623us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:59.949914 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.949974 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:00.595688 (+645714us) raft_consensus.cc:1685] finished
0709 13:04:00.595709 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.595843 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2106,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":79,"thread_start_us":224213,"threads_started":1,"wal-append.queue_time_us":277558}
W20260709 13:04:00.609683 14639 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.164s	user 0.002s	sys 0.000s
I20260709 13:04:00.611903 14622 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:00.612401 14622 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:00.613229 14622 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:00.569326 14439 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.539s	user 0.006s	sys 0.005s
I20260709 13:04:00.568423 14619 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.624157 14239 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:00.625972 14581 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:00.643779 14643 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 13:04:00.646658 14638 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 13:04:00.650940  9331 raft_consensus.cc:2368] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
I20260709 13:04:00.650592  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 975) took 1757 ms. Trace:
I20260709 13:04:00.654425  8655 rpcz_store.cc:276] 0709 13:03:58.893655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.742629 (+848974us) service_pool.cc:224] Handling call
0709 13:03:59.879474 (+136845us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.889809 (+ 10335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.650296 (+760487us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540025bf50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:00.650503 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":313938176}
W20260709 13:04:00.649694 14057 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.616s	user 0.013s	sys 0.000s
I20260709 13:04:00.661921  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1111) took 1669 ms. Trace:
I20260709 13:04:00.663717  8729 rpcz_store.cc:276] 0709 13:03:58.992165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.992285 (+   120us) service_pool.cc:224] Handling call
0709 13:03:59.033081 (+ 40796us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.216892 (+183811us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.216951 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.217016 (+    65us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:59.217076 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.661676 (+1444600us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b5400416254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:00.661724 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.661858 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":422045568}
W20260709 13:04:00.664502  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 116316us
I20260709 13:04:00.645078 14059 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.669996  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:00.630234 14439 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:00.635392 14448 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.628038 14629 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.694870 14059 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:00.695829 14059 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:04:00.696970 14640 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 13:04:00.674679 14598 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:00.697678 14598 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:00.698091 14598 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:00.698984 14619 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:00.703740  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.674969 14573 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:00.675384 14569 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:00.668519 14057 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.672125 14305 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 13:04:00.716661 14057 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:00.718264  9328 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:00.733889 14439 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:00.726100 14569 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.730151 14154 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:00.687803 14239 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:04:00.755139 14527 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:00.738932 14267 consensus_meta.cc:287] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.691s	user 0.008s	sys 0.004s
I20260709 13:04:00.770874 14448 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:00.762785  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 173934us
I20260709 13:04:00.772305  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1105) took 2058 ms. Trace:
W20260709 13:04:00.706557 14379 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.017s	user 0.006s	sys 0.013s
I20260709 13:04:00.772547 14197 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:04:00.777604 14379 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.088s	user 0.006s	sys 0.013s
I20260709 13:04:00.773697 14586 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:00.778357 14586 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:00.772727  8576 rpcz_store.cc:276] 0709 13:03:58.713717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.531207 (+817490us) service_pool.cc:224] Handling call
0709 13:03:59.559316 (+ 28109us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.559580 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:59.676047 (+116467us) log.cc:844] Serialized 14 byte log entry
0709 13:03:59.946157 (+270110us) log.cc:844] Serialized 14 byte log entry
0709 13:04:00.625897 (+679740us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b18004e0760. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:04:00.771801 (+145904us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b18004e0760. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:04:00.771949 (+   148us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:00.771994 (+    45us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:00.772044 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.772119 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.772237 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":794966,"prepare.queue_time_us":132722,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":98,"raft.queue_time_us":832440,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":188,"spinlock_wait_cycles":149182720,"thread_start_us":370699,"threads_started":2,"wal-append.queue_time_us":329194}
I20260709 13:04:00.773929 14154 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:00.786113 14154 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:00.786530 14154 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:00.774627 13948 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:00.787452 14154 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.775209 14634 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:04:00.788987 14154 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:04:00.776155  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.789312 14576 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:00.789991 14586 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:00.776897  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 97562.5us
W20260709 13:04:00.774595  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 622342us
I20260709 13:04:00.794242  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:00.795015 14527 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:00.795233 14453 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:00.795734 14453 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.795871  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1104) took 2108 ms. Trace:
I20260709 13:04:00.796276 14453 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:00.799731  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1320) took 1036 ms. Trace:
I20260709 13:04:00.800159  9322 rpcz_store.cc:276] 0709 13:03:59.763766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.763913 (+   147us) service_pool.cc:224] Handling call
0709 13:03:59.764209 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.779209 (+ 15000us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.779249 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.779300 (+    51us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:59.779338 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.799493 (+1020155us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54008f0cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:00.799543 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.799674 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33902848}
I20260709 13:04:00.796307  8584 rpcz_store.cc:276] 0709 13:03:58.687269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.418074 (+730805us) service_pool.cc:224] Handling call
0709 13:03:59.444477 (+ 26403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.444749 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:59.622380 (+177631us) log.cc:844] Serialized 14 byte log entry
0709 13:04:00.603321 (+980941us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:00.795230 (+191909us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b18001cf660. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a
0709 13:04:00.795521 (+   291us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:00.795573 (+    52us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:00.795625 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.795697 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.795818 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":225026,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":416,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":14484992,"thread_start_us":213613,"threads_started":2,"wal-append.queue_time_us":255515}
I20260709 13:04:00.780051 14267 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:00.805979 14453 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:00.808035 14267 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:00.810367  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:00.810974  8728 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:00.811551 14612 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 1.219s	user 0.005s	sys 0.008s
I20260709 13:04:00.811784 14453 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:00.795392  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:00.814051  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:00.814776  8731 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:00.818130  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:00.819443  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.820996  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.822395  9326 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:00.822726  9322 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:00.822074  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:00.820302  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.824971  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:00.825834  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:00.830559  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:00.831379  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:00.831517  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:00.832552  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.835042  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:00.836937  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:00.834280 14458 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:00.835068  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:00.824126 14572 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:00.852949  8729 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:00.843432  8728 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:00.845185 14386 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.845789 14656 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:04:00.848846 14635 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.354s	user 0.001s	sys 0.001s
I20260709 13:04:00.817461 14057 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 13:04:00.851120 13948 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:00.857738 14347 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:00.856959 14386 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:00.899580 14622 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:00.896112 14347 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:00.901389 14386 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:00.866492 14619 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:00.903403 14619 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.903730 14619 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:00.906699  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 981) took 1829 ms. Trace:
I20260709 13:04:00.906912  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:00.908725 14641 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.437s	user 0.001s	sys 0.001s
I20260709 13:04:00.909940  8731 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:00.911062  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:00.913486  8725 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:00.914870  8724 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:04:00.915283  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1071) took 1010 ms. Trace:
I20260709 13:04:00.870267 14656 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.916111 14656 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.907121  8659 rpcz_store.cc:276] 0709 13:03:59.077774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.756009 (+678235us) service_pool.cc:224] Handling call
0709 13:03:59.756371 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:59.784838 (+ 28467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.784870 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:59.785772 (+   902us) log.cc:844] Serialized 26 byte log entry
0709 13:04:00.110182 (+324410us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:00.161078 (+ 50896us) log.cc:844] Serialized 14 byte log entry
0709 13:04:00.471474 (+310396us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.471554 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:00.624643 (+153089us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b1800098fe0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b29522 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:04:00.809206 (+184563us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b48001b2358. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:04:00.848242 (+ 39036us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:00.906443 (+ 58201us) raft_consensus.cc:1685] finished
0709 13:04:00.906479 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.906645 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":56058,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":7041,"raft.queue_time_us":48180,"raft.run_cpu_time_us":4010,"raft.run_wall_time_us":627545,"spinlock_wait_cycles":106053120,"thread_start_us":631383,"threads_started":2,"wal-append.queue_time_us":708195}
I20260709 13:04:00.877087 14501 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.922189 14501 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:00.922411 14512 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:00.857976 14561 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:00.879585  8478 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 205) took 37312 ms (client timeout 15000 ms). Trace:
I20260709 13:04:00.893309  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 1316) took 1147 ms. Trace:
I20260709 13:04:00.929021  9327 rpcz_store.cc:276] 0709 13:03:59.746068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.746608 (+   540us) service_pool.cc:224] Handling call
0709 13:04:00.893066 (+1146458us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5400a06254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:00.893235 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24626048}
W20260709 13:04:00.864285  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.932355  8623 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:00.915663  9325 rpcz_store.cc:276] 0709 13:03:59.905367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.905551 (+   184us) service_pool.cc:224] Handling call
0709 13:04:00.893455 (+987904us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:00.893677 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:00.893704 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:00.893748 (+    44us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:00.893778 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.893832 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.895128 (+  1296us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b5400a06250. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:00.915231 (+ 20103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2255868800}
I20260709 13:04:00.936197  8724 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:00.938411  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:00.879669  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:04:00.954705  8652 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:00.948292  8659 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
W20260709 13:04:00.960439  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 125417us
I20260709 13:04:00.942440  9327 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:04:00.963387  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:00.972733  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:00.973497  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 1.02163e+06us
I20260709 13:04:00.970359  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:00.976514  8658 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:00.969026  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:00.969719  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:00.966202  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:00.975212  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:00.970037  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:00.987991  8656 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.968221  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:00.988857  8652 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:00.989238  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:00.990828  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:00.996863  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:00.997882  8653 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:00.997678  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:00.998390  8659 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:04:00.881687 13948 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:00.860271 14538 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.008023  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:01.008580  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:00.921597 14577 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.902344 14347 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:01.020908  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:00.974059 14453 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:01.014339  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.948076 14647 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:00.961326 14576 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:00.961839 14057 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.005548  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:00.963828 14514 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:04:00.946478 14621 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 13:04:01.004033 14634 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:00.856711 14598 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.029567 14572 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:01.073067  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:01.109468  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:01.109639 14598 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.110656  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:01.111382  8653 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.061719  8658 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:04:01.073776  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:01.114166  8654 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:01.100955  8656 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:00.995956 14644 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
W20260709 13:04:01.106647  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.998292 14388 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.068s
I20260709 13:04:00.883993 14586 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.115413  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:01.120764  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:01.126091  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 379713us
I20260709 13:04:01.124042  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:01.132475  8696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.699s (SENT) [suppressed 20 similar messages]
W20260709 13:04:01.132793  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.010172 14532 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 13:04:01.131102  8698 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:01.008163 14646 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.116s
I20260709 13:04:01.024731 14538 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.084636 14647 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:01.160501  8653 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:01.163686  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:01.027721 14577 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.171126  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:01.138902  8659 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:04:01.061432 14267 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 13:04:01.123564 14576 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:01.174643  8658 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:01.113329 14653 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 13:04:01.184931 14532 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:01.185454 14532 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:01.186133  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55137.6us
I20260709 13:04:01.114706 14540 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 13:04:01.129344 14473 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.066087 14453 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.196497  8658 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:01.148474 14442 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:04:01.203575  8657 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:01.145428 14197 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:01.105453 14057 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.199040 14379 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:01.211028 14379 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:01.211665 14379 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:00.880364  8476 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:01.220291  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:01.165112  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 50086.8us
I20260709 13:04:01.176900  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:01.165109  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:01.224066 14055 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:01.224565 14055 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:01.198376  8658 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:04:01.225461 14055 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.226401 14442 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:01.226800 14055 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:01.227347 14442 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.232782 14453 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.189278 14512 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:04:01.237936  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:01.239168  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 250010us
I20260709 13:04:01.243047  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:00.928642  8478 rpcz_store.cc:269] 0709 13:03:23.566880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.597608 (+ 30728us) service_pool.cc:224] Handling call
0709 13:04:00.879335 (+37281727us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:51.322633 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:51.323532 (+   899us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:51.323606 (+    74us) write_op.cc:435] Acquired schema lock
0709 13:03:51.323634 (+    28us) tablet.cc:662] Decoding operations
0709 13:03:51.410982 (+ 87348us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:51.419521 (+  8539us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:51.440033 (+ 20512us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:03:51.513796 (+ 73763us) tablet.cc:701] Row locks acquired
0709 13:03:51.516549 (+  2753us) write_op.cc:260] PREPARE: finished
0709 13:03:51.568480 (+ 51931us) write_op.cc:270] Start()
0709 13:03:51.595055 (+ 26575us) write_op.cc:276] Timestamp: P: 1783602231516805 usec, L: 0
0709 13:03:51.595142 (+    87us) op_driver.cc:348] REPLICATION: starting
0709 13:03:51.596696 (+  1554us) log.cc:844] Serialized 5428 byte log entry
0709 13:03:52.387627 (+790931us) op_driver.cc:464] REPLICATION: finished
0709 13:03:52.842425 (+454798us) write_op.cc:301] APPLY: starting
0709 13:03:52.842546 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:53.047883 (+205337us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:53.048025 (+   142us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:53.414261 (+366236us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:53.414622 (+   361us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 13:03:53.414685 (+    63us) write_op.cc:312] APPLY: finished
0709 13:03:53.415155 (+   470us) log.cc:844] Serialized 129 byte log entry
0709 13:03:53.486944 (+ 71789us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:53.500065 (+ 13121us) write_op.cc:454] Released schema lock
0709 13:03:53.517496 (+ 17431us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:53.605717 (+ 88221us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:53.605757 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5264907,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":441134,"apply.run_cpu_time_us":34840,"apply.run_wall_time_us":763246,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62619,"num_ops":14,"prepare.queue_time_us":552855,"prepare.run_cpu_time_us":12644,"prepare.run_wall_time_us":399288,"raft.queue_time_us":352977,"raft.run_cpu_time_us":96424,"raft.run_wall_time_us":315404,"replication_time_us":760503,"thread_start_us":997685,"threads_started":5,"wal-append.queue_time_us":373886,"wal-append.run_cpu_time_us":50014,"wal-append.run_wall_time_us":2506225}]]}
I20260709 13:04:01.196066 14573 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:01.202137 14576 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.183337 14666 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 13:04:01.234186  8476 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 7 to 9, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 9 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:01.250947  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:01.273756  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:01.274972  9296 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:01.239689 14512 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:01.208912 14473 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:01.282117  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.505s (SENT)
I20260709 13:04:01.254912 14197 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:01.290987  8653 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:01.293900  8548 leader_election.cc:400] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:04:01.297165  8548 leader_election.cc:403] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:01.313365  9331 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
W20260709 13:04:01.352535  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57334.3us
W20260709 13:04:01.358350  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 43872.4us
I20260709 13:04:01.265975 14621 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.272889 14453 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:01.360167  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:01.278844 14458 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:01.272698 14505 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.276893 14536 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.285696 14528 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.309466 14573 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:01.267270 14576 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.401919  8548 leader_election.cc:400] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
I20260709 13:04:01.438409  8548 leader_election.cc:403] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:04:01.439003  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 44576.5us
I20260709 13:04:01.439790  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:01.348589 14197 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:04:01.443198  8652 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
W20260709 13:04:01.444557  8696 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.771s (SENT)
I20260709 13:04:01.358753 14154 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:01.445966 14154 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:01.446636 14154 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:01.383505 14305 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 13:04:01.454077 14305 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:01.454604 14305 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:01.361274 14626 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:01.455968 14626 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.459080 14626 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.433079 14536 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.462213 14536 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.394928 14453 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.392658 14666 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:01.393651 14665 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:01.267897 14647 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.429737 14566 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.472720 14453 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:01.264605 14532 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:01.495335  9297 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.006s	sys 0.009s
W20260709 13:04:01.495920  9297 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.006s	sys 0.010s
W20260709 13:04:01.496690  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 670781us
I20260709 13:04:01.394462 14621 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:01.499054  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 508232us
I20260709 13:04:01.499929  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:01.393219 14656 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.477764 14644 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.427475 14505 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:01.495204 14668 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 13:04:01.434478 14458 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 13:04:01.482160 14586 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.540277  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:01.490942 14663 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:01.433943 14528 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.553306  9297 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:01.565302 14663 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.567770 14663 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.507844 14566 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.572396 14566 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.573033 14566 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.592892  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:01.573616 14586 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:04:01.562016 14659 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.357s	user 0.001s	sys 0.001s
I20260709 13:04:01.576102 14656 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.578372 14538 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.589190 14505 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.516921 14305 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.433187 14512 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.603603  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:04:01.615370  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:01.593329 14639 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 13:04:01.591907 14621 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:01.593719 14660 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.402s	user 0.000s	sys 0.002s
I20260709 13:04:01.480036 14666 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.602098 14569 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:04:01.621079  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 346020us
I20260709 13:04:01.599033 14532 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.601696 14654 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:01.591970 14528 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.661293  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:01.505997 14626 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.096s
I20260709 13:04:01.522740 14644 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.624976 14681 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:04:01.654772  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:01.656875  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:01.624100 14586 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:01.646090  9330 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:04:01.661979  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:01.616173 14653 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:01.601331 14566 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.661489 14512 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:01.676517  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.649204 14656 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:04:01.676932  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 266095us
I20260709 13:04:01.657261 14305 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:01.691453  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:01.681433 14569 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:04:01.694061 14347 consensus_meta.cc:287] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.664s	user 0.012s	sys 0.000s
I20260709 13:04:01.649879 14057 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:01.675197  9296 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:01.665509 14654 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.695618 14654 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:01.695247 14347 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.695952  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:01.699337 14057 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:04:01.699549 13948 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.673s	user 0.008s	sys 0.002s
I20260709 13:04:01.699961 14057 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:01.700098 13948 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.563509 14573 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.700924 14057 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.703478 14197 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14.
I20260709 13:04:01.664225 14621 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:01.687642 14673 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 13:04:01.683871 14686 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:01.708187 14686 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.708657 14686 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:01.709733 14653 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:01.690704 14656 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:01.710389 14653 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:01.710846 14666 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:01.711233 14656 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.711447 14653 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.681697 14473 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.694530 14569 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:01.714635 14569 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.705705 14621 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:01.717651 14664 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 13:04:01.719081 14621 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.683275  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:01.721799  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.683743 14458 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:04:01.725793 14197 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:01.726311 14197 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:01.730921 14644 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:01.731698 14619 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.828s	user 0.012s	sys 0.000s
I20260709 13:04:01.732244 14619 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:01.734874  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.730255 14197 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.735605 14059 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:01.736095 14059 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:01.738519  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:01.707666 14566 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.746304  9326 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:01.749212  9328 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 18: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:04:01.717240 14622 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.817s	user 0.007s	sys 0.003s
I20260709 13:04:01.752271  9323 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:04:01.752673 13948 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.700848 14453 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:01.749558 14685 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260709 13:04:01.749799 14671 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 13:04:01.737892 14379 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.779109 14439 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:01.779577 14439 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:01.780272 14379 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.780457 14439 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.782194 14439 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.784972 14577 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.785180 14654 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:01.787933 14670 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.297s	user 0.001s	sys 0.000s
I20260709 13:04:01.788228 14654 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.788661  9330 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:01.772362 14576 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.793708 14619 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:01.691250 14154 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.680146 14688 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 13:04:01.797118 14154 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:01.804288 14561 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.878s	user 0.004s	sys 0.009s
I20260709 13:04:01.806826 14586 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:01.811671 14561 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:01.811062 14347 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:01.821636 14453 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:01.658677  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 218972us
I20260709 13:04:01.858949  8476 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 9 to 12, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 12 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:01.871840  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 351949us
I20260709 13:04:01.873046  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.876750  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.873075  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.878556  8577 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:04:01.879031  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.880025  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.876792  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.886034  8579 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:01.888741  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.882966  8584 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:04:01.883359  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.892534  8579 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:01.892694  8582 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:01.884788  8583 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:01.893184  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:01.894747  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.895131  8583 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
W20260709 13:04:01.896238  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 543586us
I20260709 13:04:01.893544  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.905450  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.907395  8583 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:01.905203  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.882272  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.911128  8581 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.914235  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.914887  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.890420  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.917376  8577 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
W20260709 13:04:01.916541  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.891284  8476 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:01.923416  8476 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:01.924371  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.918507  8584 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:01.929788  8729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.003s	sys 0.002s
W20260709 13:04:01.930052  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:01.930433  8729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.003s	sys 0.003s
I20260709 13:04:01.930992  8730 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: []
W20260709 13:04:01.931244  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.931734  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:01.933071  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:01.943007  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.946873 14453 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:01.949519 14442 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:01.950824  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.950546 14442 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.953099  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:01.953428 14694 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:01.954150  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:01.954425 14694 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:01.955138  9322 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:01.961133  8576 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:01.964212  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:01.965109  9328 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 13:04:01.965824  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:01.966542  8725 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:01.968500  8583 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:04:01.969328  8581 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:01.944921 14703 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 13:04:01.972838  8476 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 9 to 11, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 11 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:01.978854 14294 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:01.979352 14294 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:01.978440  8476 catalog_manager.cc:5697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 10 to 11, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 11 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:01.983700  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.984831  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3183) took 1015 ms. Trace:
I20260709 13:04:01.985570  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:01.985428  8650 rpcz_store.cc:276] 0709 13:04:00.968976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.969083 (+   107us) service_pool.cc:224] Handling call
0709 13:04:01.984348 (+1015265us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b5400410854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:01.984586 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2018216320}
I20260709 13:04:01.986811  8584 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:01.988072 14619 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:01.988245  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.990397  8577 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:01.989650 14586 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 13:04:01.993818 14347 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:01.994416 14347 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:01.994731  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2915) took 1056 ms. Trace:
I20260709 13:04:01.962180  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:01.995645 14347 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:01.997058 14347 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:01.999230  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.995240  8729 rpcz_store.cc:276] 0709 13:04:00.938766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.938921 (+   155us) service_pool.cc:224] Handling call
0709 13:04:00.999712 (+ 60791us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:00.999932 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:01.199478 (+199546us) log.cc:844] Serialized 14 byte log entry
0709 13:04:01.994022 (+794544us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:01.994144 (+   122us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:01.994307 (+   163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:01.994446 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:01.994652 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38001,"prepare.queue_time_us":183200,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":61733,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":272,"thread_start_us":930318,"threads_started":2,"wal-append.queue_time_us":620307}
I20260709 13:04:01.999605  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.004431 14694 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:02.004801  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.002578  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:01.991066 14442 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:02.013621  8580 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:02.012950  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.017341  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.018571 14703 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:02.019290  8730 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:02.019924 14703 consensus_queue.cc:1243] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:04:02.020890 14703 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:02.021389 14703 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:02.023061 14703 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:02.024523 14512 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.028561 14512 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.031013  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.032136  8577 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:02.033519  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:01.989197 14619 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.045915  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.052244  8580 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.980842 14294 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:02.048674  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.046240 14619 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:02.064735  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.056360  8577 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:02.021551  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.062525  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.062932  8579 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:02.055400  8725 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:02.048441 14059 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.070399  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:02.071985 14704 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.073004  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.074692  9323 raft_consensus.cc:2410] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:04:02.075024  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3182) took 1106 ms. Trace:
I20260709 13:04:02.075544  8657 rpcz_store.cc:276] 0709 13:04:00.968699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.968809 (+   110us) service_pool.cc:224] Handling call
0709 13:04:02.074552 (+1105743us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400410ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:02.074809 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":504813440}
I20260709 13:04:02.079844  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:02.072708 14704 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.081259 14294 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:02.068931 14059 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:02.077054  8656 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:02.089174 14704 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:02.089382  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.090263  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.091660  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.095144  8581 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:04:02.095561  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.097323  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:02.097846  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 95635.5us
I20260709 13:04:02.099062 14057 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:02.099526 14055 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:02.099632 14057 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:02.100283 14055 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:02.100272 14057 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:02.103093  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:02.105938 14665 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:02.106693 14619 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:04:02.108201  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.107899 14501 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:02.091784  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.108708  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.109771  8577 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:02.109231 14055 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:02.110875  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.110962  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.111675  8577 raft_consensus.cc:2376] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17. Re-sending same reply.
I20260709 13:04:02.112500  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.112860  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.114923  8733 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:04:02.118989  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.119823 14453 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.112767  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.128047 14453 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.129230  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.130628 14453 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:02.130962  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.131651  8577 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:02.133109  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:02.135136  8724 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:02.130599  9322 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:02.135967  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:02.137727  8583 raft_consensus.cc:2376] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:04:02.138438  8732 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:04:02.127175  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.139487  8581 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:02.105651 14704 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:02.025182 14703 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:02.133042  8657 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:02.105317  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:02.134436  8654 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:04:02.031534 14512 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:02.169265  8576 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:02.168943  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:02.186314  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:04:02.189571  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 152097us
I20260709 13:04:02.133728  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:02.192835  8651 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:02.193755  8730 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:02.157900  8728 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:02.197547  8584 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:02.203238  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.204242  8581 raft_consensus.cc:2376] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14. Re-sending same reply.
I20260709 13:04:02.157315  8577 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:02.206655  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.207262  8577 raft_consensus.cc:2376] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16. Re-sending same reply.
I20260709 13:04:02.208629  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.210793  8576 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:04:02.210875  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47237.7us
I20260709 13:04:02.212343  8726 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:02.169267  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:02.234958  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.235468  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53875.6us
I20260709 13:04:02.237592 14666 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:02.238745 14666 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:04:02.239415 14666 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:02.239609 14704 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:04:02.240783 14686 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.531s	user 0.003s	sys 0.011s
I20260709 13:04:02.240888 14666 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.241520 14686 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.242448  8652 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:02.243490  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:02.244805 14686 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:02.245434  8726 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:02.246469 14666 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:02.249815  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:02.250955  8576 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:02.251777  8575 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:02.252249  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.252830  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.254987  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:02.256143 14512 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:02.251030  8728 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 18: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:02.258270  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48113.8us
I20260709 13:04:02.238646  8476 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:02.288472  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 183302us
I20260709 13:04:02.280504  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:02.281308  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.294948  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:02.295253  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.299576 14057 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.300624  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.303121  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.304082  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.304991  8582 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:02.305845  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.307044  9330 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:04:02.320473  8579 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:02.321302  8584 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:02.321738  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:02.325244  9323 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:02.325361  8579 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:02.327329  8579 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: [13.9-15.10]
I20260709 13:04:02.327533 14686 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:02.327719  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:02.328895 14686 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:02.329348 14686 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:02.329885  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.330274 14681 consensus_meta.cc:287] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.648s	user 0.012s	sys 0.000s
I20260709 13:04:02.330744  8582 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:02.331961  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.332540 14663 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:02.333232 14663 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:02.334812 14663 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.338119  9328 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:02.330663 14686 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.339222 14686 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:02.340013  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.332589  8582 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:02.341732  8724 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:04:02.343312  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:02.256916 14505 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 13:04:02.270464 14704 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:02.349712  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.351357  8583 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:04:02.349323  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.353542  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.364064  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 70199.2us
I20260709 13:04:02.367363  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.369874  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.371174  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.371778  8580 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:02.385847  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.386745  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.387290  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.387980  8582 raft_consensus.cc:2376] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11. Re-sending same reply.
I20260709 13:04:02.390482  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.391093  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.391758  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.392563  8579 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:02.391727  9327 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:02.393672  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:02.394207  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.394783  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.395380  9327 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:04:02.395416  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:02.396294  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.396796  9322 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:02.396920  9327 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:02.392403  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:02.389195  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.400928  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.330796 14681 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
I20260709 13:04:02.412524  8581 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:02.413553  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:02.414376  8581 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:04:02.418322  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45314.4us
I20260709 13:04:02.420423  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:02.421473  9328 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:02.422464  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:02.424125  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:02.425884 14573 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:02.427284  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.428136  8580 raft_consensus.cc:2376] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13. Re-sending same reply.
I20260709 13:04:02.340936  8476 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:02.432828  8476 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 4 to 10. New cstate: current_term: 10 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:02.436631  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.394243 14057 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:02.434610 14619 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:02.444332 14619 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:02.444707  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 987) took 1468 ms. Trace:
I20260709 13:04:02.444944 14619 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:02.418684  8579 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:02.419265  8581 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:02.455385  8582 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:02.435011  8476 catalog_manager.cc:5697] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 6 to 9, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 9 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:02.461038  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:02.436385 14704 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.465687 14573 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:02.466462 14573 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:02.468165 14573 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.469435  9330 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260709 13:04:02.473596  8584 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:02.474655 14197 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:02.475534  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:02.482502 14665 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:02.475759  8548 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:02.476766  9330 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:02.477449 14055 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.009s
I20260709 13:04:02.483198 14665 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:02.478827  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:02.456669  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:02.485776 14665 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:02.486847  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 77204.6us
I20260709 13:04:02.487931  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:02.487617  8476 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:02.490305  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.491281  8577 raft_consensus.cc:2376] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16. Re-sending same reply.
I20260709 13:04:02.445282  8656 rpcz_store.cc:276] 0709 13:04:00.975881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.180910 (+205029us) service_pool.cc:224] Handling call
0709 13:04:02.075940 (+895030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:02.087091 (+ 11151us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.444381 (+357290us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b5400410ad0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:02.444601 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2056914688}
I20260709 13:04:02.493534  8578 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10.
I20260709 13:04:02.494429  8576 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
W20260709 13:04:02.509943  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.514428  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.515256  8580 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:02.516012  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.514453  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.516793  8581 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16. Re-sending same reply.
I20260709 13:04:02.516793  8577 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:02.521629  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.518006  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:02.526666  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.528196  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.520608  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45004us
I20260709 13:04:02.531242  8581 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 13:04:02.532451  8583 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:02.533290  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.534633  8584 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:02.535257  8577 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:04:02.525043  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.539620  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:02.539199  8575 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:02.551787  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:02.553694 14681 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:02.554242 14681 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.554690 14681 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:02.550099  8729 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:04:02.490641  8476 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 reported cstate change: term changed from 7 to 10, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 10 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:02.562294 14267 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:02.563149  8476 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 5 to 8, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 8 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:02.565379 14442 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:02.565989 14442 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:02.567523 14305 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:02.566203  8476 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 reported cstate change: config changed from index -1 to 7, term changed from 7 to 9, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to <none>, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 9 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:04:02.568329 14305 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.529858 14686 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:02.519754 14537 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 13:04:02.603309 14453 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:02.604686 14453 consensus_queue.cc:1243] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Peer 453e644bad5d410ab9be85b353e28dc2 log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:04:02.605305 14703 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:02.578166 14057 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:02.605821 14703 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:02.606588 14057 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.607195 14703 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.608763 14703 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:02.609997 14710 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 13:04:02.573385 14347 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:02.615407 14347 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.615788 14347 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:04:02.617842  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59844.6us
I20260709 13:04:02.621559 14442 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.623734 14442 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:02.605849 14530 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:02.626513 14530 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:02.627713 14530 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.577565 14305 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:02.629717 14305 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.630436 14305 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:02.578617 14686 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:02.634895 14686 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:02.639600  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.643153  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.651471  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.652194  8578 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:02.651715  8583 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:02.655483  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:02.666597  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.671640  8582 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:04:02.672652  8581 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:04:02.677543  8583 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:04:02.697894  8578 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:02.719645  8579 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 13:04:02.720242  8579 pending_rounds.cc:85] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 11
I20260709 13:04:02.727006  8584 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:02.720585  8579 pending_rounds.cc:107] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:04:02.728686  8579 raft_consensus.cc:2889] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:02.732676  8579 pending_rounds.cc:107] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:04:02.734679  8579 raft_consensus.cc:2889] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:02.733841  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.718119 14619 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:02.812301  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.818499  8727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.022s	sys 0.075s
W20260709 13:04:02.818935  8727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.022s	sys 0.076s
I20260709 13:04:02.721422 14305 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:02.825570  8650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.010s
W20260709 13:04:02.825996  8650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.006s	sys 0.010s
I20260709 13:04:02.828370 14057 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:02.889505  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 546264us
W20260709 13:04:02.910219  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.910746  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.936128  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.940901  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.941175  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:02.941933  8584 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:02.944298  8579 raft_consensus.cc:2410] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:04:02.930245 14703 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.936025 14453 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:02.951117  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 60790.5us
I20260709 13:04:02.957010  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:02.967770  8584 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:04:02.952263 14505 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:04:02.954676 14663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.015s	sys 0.019s
W20260709 13:04:02.973321  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.991350  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 424475us
I20260709 13:04:02.990159  8584 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
W20260709 13:04:02.997094  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 477640us
W20260709 13:04:03.003134  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:03.000213  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 475114us
W20260709 13:04:03.021201  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.954068 14703 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:02.968341 14453 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:03.198194  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.073997  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:03.045298  8579 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
W20260709 13:04:03.097519  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 375684us
W20260709 13:04:03.158411  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 132442us
W20260709 13:04:03.250514  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.196662  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1002) took 1170 ms. Trace:
W20260709 13:04:03.272624  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 101692us
W20260709 13:04:03.170360  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 198040us
I20260709 13:04:03.082203  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:02.990399 14663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.015s	sys 0.019s
I20260709 13:04:02.975142 14505 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.052044  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:03.290210 14740 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 13:04:03.295775  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:03.180446 14512 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.114s
W20260709 13:04:03.130321 14681 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.004s
W20260709 13:04:03.095477 14719 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 13:04:03.290992 14530 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.264873 14197 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.308197  8580 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
W20260709 13:04:03.097801 14519 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.000s	sys 0.020s
W20260709 13:04:03.229754 14741 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 13:04:03.150681 14731 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:04:03.312809  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:03.013201 14694 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.006s	sys 0.029s
I20260709 13:04:03.097741 14508 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.297946 14505 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:03.065062 14566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.000s	sys 0.030s
I20260709 13:04:03.015421 14619 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:03.310223  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.092198 14728 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:03.212262 14537 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:03.346525  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:03.311685  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.028553 14305 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:03.088392 14055 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:03.316211  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 259579us
I20260709 13:04:03.316797  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1102) took 1009 ms. Trace:
I20260709 13:04:03.126977 14739 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:03.189177 14059 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.124s
I20260709 13:04:03.304965 14665 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 13:04:03.325678  9330 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:03.315120  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:03.327203  8726 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:04:03.371129  8726 pending_rounds.cc:85] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 11
I20260709 13:04:03.371425  8726 pending_rounds.cc:107] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:04:03.371690  8726 raft_consensus.cc:2889] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:03.280030  8650 rpcz_store.cc:276] 0709 13:04:02.026360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.026606 (+   246us) service_pool.cc:224] Handling call
0709 13:04:02.027344 (+   738us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:02.027670 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:02.211957 (+184287us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.196243 (+984286us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.196300 (+    57us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:03.196355 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.196428 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.196562 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22115,"prepare.queue_time_us":347105,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":391565,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":379,"thread_start_us":1089445,"threads_started":3,"wal-append.queue_time_us":850030}
W20260709 13:04:03.331619  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.333443  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:03.326052 14694 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.007s	sys 0.029s
I20260709 13:04:03.348138  8579 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:03.332676 14566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.989s	user 0.005s	sys 0.035s
W20260709 13:04:03.361413  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 64661.9us
I20260709 13:04:03.361704  8625 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:03.362331  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 66632.8us
I20260709 13:04:03.137677 14727 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.227s
I20260709 13:04:03.365710 14688 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:03.360862 14537 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.378571 14508 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.379194 14703 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:03.307541 14681 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.383113  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.386703  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.386719  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.386611  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:03.387974  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.388113  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.442543  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.389115  8659 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 13:04:03.389165  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.389868  8730 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:03.390080  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.389870  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:03.392298  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.361405 14197 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.457578 14197 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:03.404425 14059 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:03.458374  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:03.404811  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:03.404573 14665 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:03.459672  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:03.460325  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55588.5us
I20260709 13:04:03.460631 14665 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:03.461438 14665 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:03.459739  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:03.461798  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:03.461899 14665 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:03.462440 14294 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:03.406649  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.464017  8657 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:03.464095 14294 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:04:03.411513  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71391.9us
I20260709 13:04:03.464993  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.363191  9331 rpcz_store.cc:276] 0709 13:04:02.307293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.307451 (+   158us) service_pool.cc:224] Handling call
0709 13:04:02.307972 (+   521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:02.308362 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:02.573534 (+265172us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.228341 (+654807us) spinlock_profiling.cc:243] Waited 8.4 ms on lock 0x7b18001c9180. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9
0709 13:04:03.316350 (+ 88009us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.316424 (+    74us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:03.316487 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.316586 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.316729 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":515505,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1585,"raft.queue_time_us":51989,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":13803,"spinlock_wait_cycles":19784448,"thread_start_us":351195,"threads_started":2}
I20260709 13:04:03.465782  8657 raft_consensus.cc:2368] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
W20260709 13:04:03.420753  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.466939  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.442559  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.447346  8651 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:03.446141  8579 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:03.448180  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.454702  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3077) took 1457 ms. Trace:
I20260709 13:04:03.476408  8652 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:03.460441  8577 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:03.477521  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:03.478138  8577 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:03.466647  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:03.408409  8581 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:03.482446  8655 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:03.445636  8732 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:03.475545  8580 rpcz_store.cc:276] 0709 13:04:01.997268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.518175 (+520907us) service_pool.cc:224] Handling call
0709 13:04:02.535051 (+ 16876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.308654 (+773603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.454379 (+145725us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b54001680d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:03.454627 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1777447168}
W20260709 13:04:03.470801  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 109018us
I20260709 13:04:03.470170  8731 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:03.490635  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.491472  8650 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.492884 14508 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.438141 14703 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:03.493607 14656 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
I20260709 13:04:03.494264 14703 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.497076 14703 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.498208  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.501653  8582 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:03.504454  8651 raft_consensus.cc:2410] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:04:03.505240  8729 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:04:03.505091  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.504473  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.507891  8579 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:03.509778  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:03.510655  8583 raft_consensus.cc:2376] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 10. Re-sending same reply.
W20260709 13:04:03.512307  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.433209 14055 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.518633  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.519574  8582 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
W20260709 13:04:03.519855  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:03.521505  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.521747  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:03.524001  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.526332  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:03.526944  8651 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:03.458387 14059 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:03.532951  8658 raft_consensus.cc:2368] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
I20260709 13:04:03.533824  8652 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260709 13:04:03.535408  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.537065  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:03.537721  8581 raft_consensus.cc:2376] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16. Re-sending same reply.
I20260709 13:04:03.538499  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:03.540192  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.541433  8582 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:03.539065  8579 raft_consensus.cc:2376] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:04:03.548820  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.549698  8583 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:03.444039 14305 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.548064  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.558372  8580 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:03.566046  8659 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:04:03.566233  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:03.569064  8726 raft_consensus.cc:2410] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260709 13:04:03.537969 14686 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.902s	user 0.013s	sys 0.003s
I20260709 13:04:03.570735  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:03.572086  8581 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260709 13:04:03.573606 14747 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:04:03.573176 14059 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.574708 14059 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:03.575452 14722 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 13:04:03.576568 14738 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
I20260709 13:04:03.578855 14055 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:03.580080  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:03.581720 14267 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.581902  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:03.582341 14267 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.583467  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:03.584798 14267 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.498916 14688 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 13:04:03.530273 14750 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:03.588912  8577 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:03.590744  8577 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:04:03.592290  8577 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:03.463238 14665 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.585911  8582 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:03.571864  8579 raft_consensus.cc:2376] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14. Re-sending same reply.
I20260709 13:04:03.608217  8581 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:03.527536 14656 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 13:04:03.539986 14519 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.616272 14519 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.615407 14505 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.586056 14267 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:03.612109  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:03.621920 14505 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:03.622231 14619 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:03.622713 14505 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:04:03.623339 14267 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:03.623831  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:03.623952 14267 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:03.624127 14519 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.603307  8577 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 13:04:03.566320 14347 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.944s	user 0.010s	sys 0.006s
I20260709 13:04:03.624965 14267 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.586263 14727 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.068s
I20260709 13:04:03.586001  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:03.581758 14743 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.312s	user 0.000s	sys 0.001s
I20260709 13:04:03.613340  8580 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 13:04:03.614135 14716 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.315s	user 0.000s	sys 0.002s
I20260709 13:04:03.589802 14686 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.615368 14728 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:03.615625 14305 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:03.685389  8582 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:03.634074  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1312) took 1205 ms. Trace:
I20260709 13:04:03.712821  8650 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:03.734463  8583 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:03.625105 14519 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:03.642494  8657 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:04:03.636116  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:03.632375  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.625259  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:03.754674  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:03.755580  9327 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:03.634335 14732 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.753968  9328 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:03.738333  9326 rpcz_store.cc:276] 0709 13:04:02.380815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.381599 (+   784us) service_pool.cc:224] Handling call
0709 13:04:02.382164 (+   565us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:02.382793 (+   629us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:02.382832 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.383600 (+   768us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.816850 (+433250us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:02.816891 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.816954 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.585524 (+768570us) raft_consensus.cc:1685] finished
0709 13:04:03.585552 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.585679 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1781,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":125,"thread_start_us":428123,"threads_started":1,"wal-append.queue_time_us":904033}
I20260709 13:04:03.608208 14750 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.745927  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:03.639348 14749 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 13:04:03.655200 14197 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.767072 14197 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.626945 14622 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:03.769609 14622 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:03.770212 14728 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:03.770780 14728 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:03.770969 14622 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.771891 14728 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.773177 14305 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:03.773682 14305 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:03.774055 14305 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:03.774937 14686 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.635125 14681 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.746042 14665 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:03.778028 14197 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:03.638286 14752 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.292s
I20260709 13:04:03.654747 14663 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:03.781929 14663 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:03.784003  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.626116 14537 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.627945 14694 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.130s
I20260709 13:04:03.785427 14750 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:03.786936  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 161728us
I20260709 13:04:03.787962  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:03.634411 14748 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.428s
I20260709 13:04:03.745219  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:03.638419 14734 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:03.792483 14734 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.634202 14744 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.792987  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1137) took 1884 ms. Trace:
I20260709 13:04:03.793495  8727 rpcz_store.cc:276] 0709 13:04:01.908937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.938354 (+ 29417us) service_pool.cc:224] Handling call
0709 13:04:02.103723 (+165369us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1eee 0000000000375939 00007f140eb651e0 00007f140ec2e0c4 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:04:02.103956 (+   233us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:02.104833 (+   877us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:02.104874 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.119810 (+ 14936us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.819274 (+699464us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:02.819318 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.819380 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.792638 (+973258us) raft_consensus.cc:1685] finished
0709 13:04:03.792661 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.792914 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1090,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":162,"spinlock_wait_cycles":5170048,"thread_start_us":699021,"threads_started":1,"wal-append.queue_time_us":1082374}
I20260709 13:04:03.793378 14744 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.661324 14453 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:03.772585 14622 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:03.656163 14057 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:03.789592 14537 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:03.638065 14704 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.737748 14347 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.631031 14505 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.176s
I20260709 13:04:03.638187 14745 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.516s
I20260709 13:04:03.633847 14723 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.251s
I20260709 13:04:03.810017  9327 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:03.793005 14734 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:03.806151 14057 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:03.817526 14704 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.806347 14732 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:03.822382  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:03.834594  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 403664us
I20260709 13:04:03.882588  8576 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:04:03.886615  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 97183.7us
I20260709 13:04:03.818964 14059 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:03.903354  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 473643us
W20260709 13:04:03.923832  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.888249 14453 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:03.929380  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 461471us
I20260709 13:04:03.945873 14347 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:03.948199 14728 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:03.948621 14732 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:03.950896  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 205077us
W20260709 13:04:03.949755  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:03.959920  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.932402 14057 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:03.980871  8623 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:03.990478 14704 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:03.991257  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.992473  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:03.990958  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:03.999634  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:04.000528  8726 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:03.978569 14656 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:04.003979 14656 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.004922  8654 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:04.005445  8654 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:04.006510 14656 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.006338  8654 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.007932  8654 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:03.995731 14732 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:04.009779 14732 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:04.010145 14732 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:03.995412 14745 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:04.012820  9297 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:04.014617  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:03.983511 14688 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 13:04:04.015069  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:04.016848 14688 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:04.017715 14704 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:04.018162 14754 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:04.016758  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:03.959329 14453 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:03.981889 14663 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.020509  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:03.984712 14744 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.025045  8623 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:04.028733  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:04.025218 14665 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:04.022243 14666 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:04:04.032644  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.002s	sys 0.014s
I20260709 13:04:04.027706  8731 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:04.023175 14688 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:04.033090  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.002s	sys 0.014s
W20260709 13:04:04.058369  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:04.061627  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.068560  8582 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:04:04.070125  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.031397 14665 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:04.072480 14294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.005s	sys 0.010s
I20260709 13:04:04.072834 14744 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:04.068900 14347 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.058130 14688 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:04.075584  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:04.068693 14748 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:04:04.072402  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 487080us
I20260709 13:04:04.018580 14734 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:04.090590  8575 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.008s	sys 0.015s
W20260709 13:04:04.079447 14294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.006s	sys 0.011s
I20260709 13:04:04.080703  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:04.069417 14663 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.078522 14665 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:04.103880  8575 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.008s	sys 0.015s
W20260709 13:04:04.100127  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.115253  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.113555  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:04.114662  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:04.103008  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:04.101054  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:04.100531  8576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1143) took 1446 ms (client timeout 1618 ms). Trace:
I20260709 13:04:04.111340 14739 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:04.139050 14739 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.139438 14739 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:04.138345  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:04.138078  8576 rpcz_store.cc:269] 0709 13:04:02.642684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.642826 (+   142us) service_pool.cc:224] Handling call
0709 13:04:04.088353 (+1445527us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540002dd54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:04.088535 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2847663872}
I20260709 13:04:04.146184  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:04.156045  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.158164  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.100736 14748 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:04.188452  8651 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:04.153573  8724 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:04.195920  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:04.190222  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:04.191658  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.199527  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.170698  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.106693 14294 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:04.091616 14745 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 13:04:04.162555  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.165132  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:04.200448  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.103646 14347 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:04.170387 14686 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.204491  8653 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
W20260709 13:04:04.205602  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64749.3us
I20260709 13:04:04.214345 14681 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:04.217336 14681 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.147066 14663 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:04.199187 14656 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:04.164000 14744 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:04.219810  8657 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:04.215108  8658 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
W20260709 13:04:04.234800  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 116936us
I20260709 13:04:04.220295  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:04.201737 14666 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:04:04.211511 14305 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.257071 14748 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:04.260692 14734 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.242551 14197 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:04.248858 14656 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:04.190064 14756 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:04.216979 14765 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:04.279572 14765 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.281349 14537 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:04.248405 14745 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:04.283027 14745 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.283402 14745 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:04.254609 14785 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:04.285585 14267 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:04.285243 14294 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.254150 14771 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:04.295980  8656 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:04.243330 14686 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:04.293349 14771 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:04.302421 14771 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.201733  8733 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
W20260709 13:04:04.342260  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 185933us
I20260709 13:04:04.287009 14734 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:04.343539  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 273253us
W20260709 13:04:04.344069  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 321411us
W20260709 13:04:04.347544  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 232920us
I20260709 13:04:04.344476  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.349174 14773 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:04.350284  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:04.343212  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.270901 14744 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:04.366729  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:04.373747  8730 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:04.346644 14771 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:04.378859  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 263654us
I20260709 13:04:04.351446 14748 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:04.386184  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:04.400715  8657 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:04.405637  8549 leader_election.cc:400] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:04:04.372035 14773 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:04.406024  8549 leader_election.cc:403] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:04:04.406885  8653 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:04:04.384011 14305 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:04.388506 14197 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:04.403609 14750 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.614s	user 0.013s	sys 0.013s
I20260709 13:04:04.398978 14347 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.357350 14734 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:04.429102  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:04.430964  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45564us
I20260709 13:04:04.432159  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:04.420288 14057 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:04.410935 14773 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:04.415103 14759 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:04.449065  8653 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 13:04:04.429147 14453 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.459358  8654 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:04:04.412791 14769 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:04:04.459968  8654 pending_rounds.cc:85] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 11
W20260709 13:04:04.451529  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 71610.4us
I20260709 13:04:04.485737  8654 pending_rounds.cc:107] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 13.12
W20260709 13:04:04.467324  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.423080 14750 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:04.486089  8654 raft_consensus.cc:2889] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:04.492678  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1020) took 1099 ms. Trace:
I20260709 13:04:04.447885 14759 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:04.447953 14773 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:04.451009 14766 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.495152  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:04.498912  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 97438.9us
I20260709 13:04:04.499477 14766 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.501902 14766 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:04.502669  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.503902  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.504655 14347 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:04.505388 14347 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:04.505825 14453 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:04.508548  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.509476 14453 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:04.511174  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:04.512055 14566 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:04.516412  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:04.463111 14756 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:04.498061  8654 rpcz_store.cc:276] 0709 13:04:03.393569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.517055 (+123486us) service_pool.cc:224] Handling call
0709 13:04:03.541152 (+ 24097us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:04.004813 (+463661us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54003ab720. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:04:04.492305 (+487492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.492574 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":730955264}
I20260709 13:04:04.458174 14686 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:04.525081  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.413514 14656 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:04.542009 14728 consensus_meta.cc:287] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.005s
W20260709 13:04:04.493947 14512 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 13:04:04.516271 14759 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:04.547956  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.552520  8658 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
W20260709 13:04:04.552809  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 339762us
W20260709 13:04:04.541743 14732 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.531s	user 0.008s	sys 0.007s
I20260709 13:04:04.441279 14197 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.554781  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:04.556545  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48098.2us
I20260709 13:04:04.540938 14761 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:04:04.544127 14622 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.734s	user 0.008s	sys 0.008s
I20260709 13:04:04.576473  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.578290  8654 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:04.575667  8582 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:04.580201  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:04.578663  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.578135  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.585917  8659 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:04:04.495080 14786 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 13:04:04.552349 14769 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:04.589304  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 76010us
I20260709 13:04:04.584151 14753 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:04.590291 14059 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:04.590788 14059 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.591225 14059 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:04.596416 14057 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:04.599503  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.580828 14761 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:04.604768 14793 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
W20260709 13:04:04.608039 14794 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.098s	user 0.000s	sys 0.002s
I20260709 13:04:04.624806  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:04.588745 14780 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
W20260709 13:04:04.600837 14779 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.338s	user 0.002s	sys 0.001s
I20260709 13:04:04.632705  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:04.576208 14732 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:04.638041 14763 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:04.645316  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2851) took 1254 ms. Trace:
I20260709 13:04:04.590104 14294 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:04.646616  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:04.645759 14782 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.117s	user 0.000s	sys 0.003s
W20260709 13:04:04.658914  8731 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.599s	user 0.014s	sys 0.009s
I20260709 13:04:04.666950 14786 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:04.667155  8731 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:04.667583 14786 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.668923  8625 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:04.671988 14453 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:04.555265 14739 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.662163  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:04.674232  8652 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:04:04.675992  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.676934  8582 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:04.676931  8653 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:04.624477 14347 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:04:04.678736  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.680330  8733 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:04:04.680375  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:04.683274  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:04.628018 14761 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:04.690227 14057 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:04.691942  8727 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:04.653894  8655 rpcz_store.cc:276] 0709 13:04:03.390703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.506637 (+115934us) service_pool.cc:224] Handling call
0709 13:04:03.527963 (+ 21326us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:03.570729 (+ 42766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.570761 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:03.616459 (+ 45698us) log.cc:844] Serialized 52 byte log entry
0709 13:04:03.654457 (+ 37998us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b1800390640. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:03.994992 (+340535us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:04.144391 (+149399us) log.cc:844] Serialized 14 byte log entry
0709 13:04:04.144828 (+   437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.144933 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:04.642482 (+497549us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b1800390640. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1
0709 13:04:04.645093 (+  2611us) raft_consensus.cc:1685] finished
0709 13:04:04.645118 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.645258 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5037,"prepare.queue_time_us":224969,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":55285,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":6476,"spinlock_wait_cycles":38514816,"thread_start_us":457633,"threads_started":2,"wal-append.queue_time_us":528676}
W20260709 13:04:04.659974 14791 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
W20260709 13:04:04.695559  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 69898.2us
I20260709 13:04:04.695616  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:04.658180 14519 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.680095 14305 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.707032 14057 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:04.708217  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:04.709462  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84736.3us
I20260709 13:04:04.710358 14786 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.710801 14760 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:04.710970 14734 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:04.684128 14512 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:04.642323 14785 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 13:04:04.598623 14769 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:04.704929 14789 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 13:04:04.684540 14744 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.694042 14453 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:04.638039 14764 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 13:04:04.720019  8549 leader_election.cc:400] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 9 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 9. Current term is 10.
W20260709 13:04:04.728152  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 97390us
I20260709 13:04:04.733182  8730 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:04.712209 14442 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:04.727914 14760 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:04.738067  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:04.746887  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 105387us
I20260709 13:04:04.681816 14294 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.752002  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.745739  8655 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:04.754416  8625 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:04.754272  8652 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:04.756239  8653 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:04.762502  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.762880  8548 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:04.676223 14763 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:04.770953  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62824.3us
I20260709 13:04:04.771106  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.772076  8657 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 13:04:04.739475 14453 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.717316 14519 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.770388  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:04.776793  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.802506  8653 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
W20260709 13:04:04.778144 14798 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:04:04.778738 14770 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:04:04.780117  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65133.6us
I20260709 13:04:04.782652 14769 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:04.783030 14784 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:04:04.785964 14057 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:04.719772 14305 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.799706  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.805895  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:04.781277 14665 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.627s	user 0.004s	sys 0.008s
I20260709 13:04:04.830399  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:04.831596  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:04.771978 14744 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:04.844729  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:04.832804 14305 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:04.833288  8655 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:04:04.775929 14734 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:04.798454 14442 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:04.773494 14750 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.843259  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:04.844467  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71147.4us
I20260709 13:04:04.845518  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:04.850059 14809 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:04:04.850771 14770 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:04.851258  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:04.854794  9323 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:04.799155 14453 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:04.855876  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.754733 14786 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 13:04:04.865744  9326 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 13:04:04.871812  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 167097us
W20260709 13:04:04.876447  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55674.7us
I20260709 13:04:04.884120  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:04.879546  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 109241us
I20260709 13:04:04.848366 14197 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:04.884697  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63463.8us
I20260709 13:04:04.895084  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:04.897294  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.897990  8655 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:04:04.877233 14732 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:04.853910 14508 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 13:04:04.852157 14665 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.843606 14801 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 13:04:04.911504  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:04.868517 14760 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:04.903983 14803 log.cc:927] Time spent T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
W20260709 13:04:04.903817 14788 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.319s	user 0.000s	sys 0.001s
I20260709 13:04:04.917889  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.918684  8582 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:04.888095 14770 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:04.929152  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45134.1us
I20260709 13:04:04.899559 14197 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:04.942250  8655 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:04.946456  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:04.948338  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54415.4us
I20260709 13:04:04.886421 14059 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.896733 14750 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:04:04.953235  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.904119 14267 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 13:04:04.920923 14663 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:04.956285  8659 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:04.961253  9322 raft_consensus.cc:2368] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:04:04.953599 14764 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.960613  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:04.943858 14732 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
W20260709 13:04:04.973088  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 69567.7us
I20260709 13:04:04.917697 14773 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:04.972386  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:04.949844 14785 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
W20260709 13:04:04.978758 14745 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.695s	user 0.007s	sys 0.009s
I20260709 13:04:04.986732 14808 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:04.988325 14790 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
I20260709 13:04:04.993744  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:04.995249 14792 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.544s	user 0.003s	sys 0.000s
I20260709 13:04:04.995512  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.986471 14745 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:04.998215 14799 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
I20260709 13:04:04.977804 14197 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:04.996994  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:04.999020 14800 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 13:04:04.955094 14442 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.070555  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1162) took 1021 ms. Trace:
W20260709 13:04:05.071273  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.071125  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.103098  8655 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:05.102062  8577 rpcz_store.cc:276] 0709 13:04:04.049093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.049217 (+   124us) service_pool.cc:224] Handling call
0709 13:04:04.049657 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.075003 (+ 25346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.075045 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:04.075484 (+   439us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.431438 (+355954us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:04.431562 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.431650 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:05.000916 (+569266us) raft_consensus.cc:1685] finished
0709 13:04:05.070306 (+ 69390us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.070489 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36655,"raft.queue_time_us":7617,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":13141,"thread_start_us":282831,"threads_started":1,"wal-append.queue_time_us":408089}
I20260709 13:04:04.997540 14733 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 13:04:05.073347  8659 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:05.137125  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:04.997570 14810 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:05.086212  9329 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:05.094085  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 134005us
I20260709 13:04:05.088232  8581 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:04.998621 14815 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:04.969676 14663 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:04:05.104417  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 153536us
I20260709 13:04:05.132390  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1159) took 1050 ms. Trace:
I20260709 13:04:05.126370 14267 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:04:05.135331  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 188914us
I20260709 13:04:05.150724  8583 rpcz_store.cc:276] 0709 13:04:04.047874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.048027 (+   153us) service_pool.cc:224] Handling call
0709 13:04:04.070742 (+ 22715us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.103317 (+ 32575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.103368 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:04.142047 (+ 38679us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.352456 (+210409us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:04.352497 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.352570 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:04.994368 (+641798us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:05.070736 (+ 76368us) raft_consensus.cc:1685] finished
0709 13:04:05.070760 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.098014 (+ 27254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44153,"raft.queue_time_us":19636,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":4691,"spinlock_wait_cycles":5836928,"thread_start_us":147890,"threads_started":1,"wal-append.queue_time_us":219755}
I20260709 13:04:05.160102  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:04:05.131736  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 185957us
I20260709 13:04:05.087365 14305 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:05.071564 14686 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.146224  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:05.155576  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.978504 14764 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.161363  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.163542  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:05.165235  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84476.6us
I20260709 13:04:05.171617  8730 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:05.156418  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.175028  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:05.172178  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 192978us
I20260709 13:04:05.097838 14694 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:04:05.179553  8651 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:05.133543 14728 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 13:04:05.179844  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:05.188436  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:05.190058  8727 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:05.189527  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:05.163654 14267 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:05.147794 14763 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.146427 14815 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:05.160928 14294 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.162714 14665 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:05.203001  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:05.207537  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 210623us
I20260709 13:04:05.170702 14519 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.215723  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.215914  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:05.213915  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:05.217305  8548 leader_election.cc:400] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:04:05.217666  8548 leader_election.cc:403] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:04:05.203540 14663 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:04:05.210207 14267 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:05.199784 14809 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.206688 14059 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:05.225629  8654 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:05.211499 14744 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.229264 14519 raft_consensus.cc:2764] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won
I20260709 13:04:05.176975 14686 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.190672 14761 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.224879 14663 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:05.234788 14688 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.124s	user 0.015s	sys 0.010s
I20260709 13:04:05.233898 14819 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.240017 14728 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:05.220315 14294 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 13:04:05.240732 14745 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:05.298856  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 85058.6us
I20260709 13:04:05.185572 14197 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.283624 14059 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:05.250677 14508 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:05.243963 14810 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:05.299546 14663 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:05.305019 14827 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:05.305157 14824 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.304737 14830 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:05.305770 14813 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
W20260709 13:04:05.306303 14811 log.cc:927] Time spent T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 13:04:05.313794 14819 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.364924 14819 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.252130 14744 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.365229 14830 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.318686 14059 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:05.366840 14059 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:05.367300 14744 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.367628 14830 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:05.306412 14796 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.701s	user 0.000s	sys 0.005s
I20260709 13:04:05.309262 14294 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:05.348830 14827 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:05.371011 14294 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:05.372020 14827 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.330758 14831 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:05.373431  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2512) took 2355 ms. Trace:
I20260709 13:04:05.311527 14814 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:05.313118 14820 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:05.375891 14820 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:05.376778 14512 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.628s	user 0.009s	sys 0.000s
I20260709 13:04:05.376823 14305 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:05.313781 14773 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.378496 14773 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:05.303610 14833 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.379705 14833 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.382022 14833 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.173597 14770 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:05.384763 14756 consensus_meta.cc:287] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.843s	user 0.010s	sys 0.000s
I20260709 13:04:05.303707 14817 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.133s
W20260709 13:04:05.385690  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 83112.6us
I20260709 13:04:05.359166  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:05.386404  8724 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:05.300567 14815 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.386829  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:05.387689 14815 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:05.388026  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:05.360066  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:05.360164  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:05.361109 14835 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.361557 14686 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.361718 14824 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.362178  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:05.344750 14771 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.960s	user 0.007s	sys 0.009s
I20260709 13:04:05.300190 14728 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:05.374006  8575 rpcz_store.cc:276] 0709 13:04:03.017901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.582872 (+564971us) service_pool.cc:224] Handling call
0709 13:04:03.583182 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.133474 (+550292us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:04.354374 (+220900us) log.cc:844] Serialized 14 byte log entry
0709 13:04:04.417597 (+ 63223us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:04.436291 (+ 18694us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.448398 (+ 12107us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:04.448435 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.448503 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:05.305977 (+857474us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b240011e748. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:04:05.373076 (+ 67099us) raft_consensus.cc:1685] finished
0709 13:04:05.373194 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.373348 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5099,"prepare.queue_time_us":421252,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":71352,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":10603,"spinlock_wait_cycles":59240320,"thread_start_us":788425,"threads_started":3,"wal-append.queue_time_us":182871}
I20260709 13:04:05.378530 14820 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.389752  8654 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.319132 14750 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.394518  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:05.398486  8651 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:05.373143 14827 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:05.390278  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 149100us
I20260709 13:04:05.389374  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:05.385272 14756 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:05.405169  8730 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
W20260709 13:04:05.410221  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 221834us
I20260709 13:04:05.319111 14508 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:05.377312 14512 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:05.406054  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:05.397646 14734 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.006s	sys 0.007s
I20260709 13:04:05.339041 14765 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.413542  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.251921 14809 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.337327 14681 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.326308 14810 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.430090  9329 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:05.408355  8655 raft_consensus.cc:2410] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:04:05.444584  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:05.401772 14305 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:04:05.396186 14566 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:05.403205 14686 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:05.344569 14453 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.461898 14836 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.462555 14836 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.464169 14453 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.447266  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:05.464753 14681 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:05.464951 14836 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.303153 14688 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.448951  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.466403  9327 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:05.466331  8650 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:05.466910 14824 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.467285 14688 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.402745 14835 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.467797  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.468695  8650 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:05.409173 14733 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:05.405261 14764 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:05.469652 14733 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:05.442927 14734 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.006s	sys 0.007s
I20260709 13:04:05.470325 14734 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:05.472278 14814 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:04:05.472060 14734 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.474406  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:05.474573  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:05.475126  9324 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:05.477035  9329 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:04:05.452075 14566 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:05.478083 14665 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:05.478312 14566 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.478972 14665 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.455886 14442 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:05.480607 14442 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:05.429138 14508 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:05.481834 14442 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.451685 14810 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:05.484282  8623 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:05.485615  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:05.486250 14761 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.486810  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:05.486900 14759 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.910s	user 0.010s	sys 0.003s
I20260709 13:04:05.455376 14686 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:05.487593 14759 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.459573  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:05.490459  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.491765  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.492414  8653 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:05.492785  8724 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:05.372020 14294 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.494374 14294 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:05.465387 14681 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.454718  9323 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:05.470163 14733 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:05.479835 14566 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:05.493322  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.506959  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:05.508666  8653 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:05.508574  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 188727us
W20260709 13:04:05.509294  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:05.504374  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 303611us
I20260709 13:04:05.509059  8654 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:05.510485  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:05.510485  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:05.511786  8651 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:05.512172  8727 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:05.508080 14756 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:05.486907 14761 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:05.415359 14750 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.516225  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:05.469256 14835 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:05.521144  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:05.521185  9329 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:05.523653  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:05.525425 14681 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.525643  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.525868 14809 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.526008 14739 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.526438 14728 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.527887 14728 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:04:05.528316 14795 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.027s	user 0.006s	sys 0.000s
I20260709 13:04:05.524080 14814 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.529443 14814 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.530021  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:05.530660  9323 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:05.532163 14814 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.532908 14663 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.534639 14750 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:05.536757 14837 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
W20260709 13:04:05.537882 14805 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.311s	user 0.002s	sys 0.000s
I20260709 13:04:05.540756  8656 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:05.514721 14756 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.541762 14756 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.518141 14761 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:05.543960 14761 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:05.546396 14763 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:05.546900 14763 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:05.547868 14763 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:05.548538  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 159253us
I20260709 13:04:05.549683 14826 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:04:05.549877 14305 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:05.551478 14838 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:05.551949 14838 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:05.552500 14059 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.553433  8696 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:05.554270 14816 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:05.557894  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.552971 14838 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.561754 14838 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:05.567169  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.578759  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:05.584709  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 73115.5us
W20260709 13:04:05.591020  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66067.4us
I20260709 13:04:05.535948 14663 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.596428  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:05.603356 14519 raft_consensus.cc:2764] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election decision vote started in defunct term 14: won
I20260709 13:04:05.604272 14519 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:04:05.606678 14519 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:05.576537 14059 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.608494  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:05.601183 14748 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:05.620227  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.621176  8578 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:05.624562  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:05.628710 14849 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.190s	user 0.003s	sys 0.000s
I20260709 13:04:05.630069 14733 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.632710 14733 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.632908  8575 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:05.633857 14759 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:05.635102  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44795.4us
I20260709 13:04:05.597685 14765 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.636399 14765 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.589680 14732 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:05.595657 14826 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.639631  8578 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:05.645743  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.648392  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.649185  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:05.650202  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.650833  8583 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:05.641577  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:05.654093  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.656898  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.658283  8575 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:05.655993  8583 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.662096  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:05.639102 14827 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:05.618417 14748 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.665117  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.607479 14267 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.643308 14732 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.610069 14508 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.689646 14508 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:05.692371  8733 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:05.651752 14665 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:05.652343 14754 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.014s	sys 0.007s
I20260709 13:04:05.644750 14770 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.680496 14732 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.684031  8732 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:05.685438 14267 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:05.677769  8578 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:05.697856  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:05.701486  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1158) took 2234 ms. Trace:
I20260709 13:04:05.674273 14827 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:05.665028 14769 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:05.683032 14765 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.710161  8579 rpcz_store.cc:276] 0709 13:04:03.466839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.609205 (+142366us) service_pool.cc:224] Handling call
0709 13:04:03.609579 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.073331 (+463752us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.073364 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:04.078090 (+  4726us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.162190 (+ 84100us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:04.162573 (+   383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.162658 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:05.699067 (+1536409us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b240011e748. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b67fb0 00007f14097de488 00007f14097debe8 00007f14097b6398 00007f14097b176a 00007f14097b0bb6
0709 13:04:05.701131 (+  2064us) raft_consensus.cc:1685] finished
0709 13:04:05.701152 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:05.701422 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":45783,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":125,"spinlock_wait_cycles":52676480,"thread_start_us":512684,"threads_started":2,"wal-append.queue_time_us":342580}
I20260709 13:04:05.679960 14748 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.727560  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.727766  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1166) took 1130 ms. Trace:
I20260709 13:04:05.728241  8581 rpcz_store.cc:276] 0709 13:04:04.597608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.598398 (+   790us) service_pool.cc:224] Handling call
0709 13:04:04.624479 (+ 26081us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.143716 (+519237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.727507 (+583791us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b540017a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:05.727700 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":859638144}
I20260709 13:04:05.728868  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.729555  8579 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.729594  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.730394  8581 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:05.731297  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.731554 14763 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:05.731962  8581 raft_consensus.cc:2368] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16.
I20260709 13:04:05.718444 14666 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:05.704909 14754 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.014s	sys 0.008s
I20260709 13:04:05.722932  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.754196  8578 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:05.719327 14769 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.717574  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:05.652215 14826 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.738741  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.716301 14827 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:05.652483 14810 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.707082 14508 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 13:04:05.748310  8579 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:05.751159  8583 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:05.753136  8575 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:05.755909 14666 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.757153  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.758028  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.770365  8578 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:05.770596  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.761580  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.772817  8576 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:05.774073  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.775244  8576 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 17 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:04:05.740865 14748 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.755049  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.761058 14769 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:05.782616  9322 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:05.779413  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.710409 14770 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:05.787551 14305 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:04:05.762717 14519 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:04:05.776728 14762 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 13:04:05.795780 14831 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:05.796162 14831 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:05.796810 14810 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.797060 14831 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.798365 14831 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:05.801903 14686 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.803710 14686 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.767007  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.784212  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.810935  8584 raft_consensus.cc:2410] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:04:05.812225 14756 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.766281 14665 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:05.813791 14665 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:05.814814 14665 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.786420  8576 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:05.786604  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:05.777448 14666 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.818888 14826 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:05.821017 14656 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:05.821441 14656 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.821816 14656 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:05.821547 14761 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.824637 14761 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.832446  8583 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:05.785281  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:05.768581  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.835482  8575 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:05.837285  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.838090  8579 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
W20260709 13:04:05.849510 14863 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:04:05.839362 14508 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:05.866909 14666 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:05.867515 14508 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:05.871194  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 86024.5us
I20260709 13:04:05.876546  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:05.872080  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:05.879606  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:05.893576  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:05.918438  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.852365 14748 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:05.922685  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:05.928457  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.928294  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:05.930660  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:05.931725  8733 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:05.929344  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:05.925451 14808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.004s	sys 0.004s
I20260709 13:04:05.957024  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:05.913985 14508 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.960147  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:05.953083  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:05.956951  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:05.919824 14688 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:06.028036  8653 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:05.961999  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:05.963622  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:06.030380  8655 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:05.965127  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:06.034161  8651 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:06.034410  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.972782  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:06.028124  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:06.002555  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.037295  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:06.037837  9330 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.038249  8583 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:06.035482  8654 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:05.968677  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:06.055058  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:06.056638  8656 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:06.056959  8653 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:06.055089  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:05.933355 14305 raft_consensus.cc:2764] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:04:06.054706  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:06.070246  8650 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:05.960748 14733 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:06.069540  8654 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:04:06.070474  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:05.927448 14519 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:04:05.961170 14808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.004s	sys 0.005s
I20260709 13:04:06.032351 14665 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.920360 14442 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.115s
I20260709 13:04:05.961433 14748 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.949877 14801 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.929157 14838 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:05.949249 14267 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.145s
I20260709 13:04:06.094347  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.076152 14733 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.175509  8732 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:06.169507  8583 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:06.169653  9327 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:06.181854  8729 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:04:06.087152 14786 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.079538 14734 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.180218  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:06.192432  8623 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.149978 14681 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:06.158272 14666 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:06.156062 14688 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:06.081840 14656 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.208163  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:06.170259 14759 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.162201 14838 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.166173 14519 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:06.162286 14859 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.216090  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:06.216991  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.181627 14733 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:06.194613 14812 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:06.241658  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:06.243101  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62964.8us
I20260709 13:04:06.152447 14665 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:06.242400 14865 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
I20260709 13:04:06.212802 14728 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:06.241562 14859 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:06.219875 14862 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 13:04:06.243510 14057 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:06.185425 14801 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.272027  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:06.183539 14748 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:06.205422 14267 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:06.262787 14057 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:06.275610 14057 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.234334 14838 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:06.232004 14519 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.271950  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:06.202942 14505 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.196836 14786 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:06.256942 14867 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:04:06.281225  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:06.213743 14769 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.289752  9328 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:04:06.193774 14566 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.299288 14566 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:06.290267  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.182991 14827 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.231746 14681 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.303359 14827 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.303901 14681 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.305486  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:06.307840 14851 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.868s	user 0.004s	sys 0.000s
I20260709 13:04:06.291586 14765 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:06.308467 14765 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.308825 14765 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:06.307337 14267 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.315794 14786 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:06.316061 14267 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.204761 14734 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.318544  8658 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:06.234095 14759 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.273423 14728 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:06.319617  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.341408  8650 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:06.342617  8581 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:06.343281  8696 leader_election.cc:400] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16.
I20260709 13:04:06.343665  8696 leader_election.cc:403] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:06.253577 14812 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.323815  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1032) took 1190 ms. Trace:
I20260709 13:04:06.288275 14519 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.344898  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:06.369313  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:06.371322  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:06.368129  8657 rpcz_store.cc:276] 0709 13:04:05.133782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.133995 (+   213us) service_pool.cc:224] Handling call
0709 13:04:05.155602 (+ 21607us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:05.188457 (+ 32855us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.188492 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:05.224231 (+ 35739us) log.cc:844] Serialized 52 byte log entry
0709 13:04:05.439320 (+215089us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:05.609121 (+169801us) log.cc:844] Serialized 14 byte log entry
0709 13:04:05.612997 (+  3876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.613165 (+   168us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.291139 (+677974us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:06.317701 (+ 26562us) raft_consensus.cc:1685] finished
0709 13:04:06.323605 (+  5904us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.323757 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14409,"prepare.queue_time_us":127670,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":47,"raft.queue_time_us":52831,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":1127,"spinlock_wait_cycles":11267328,"thread_start_us":379060,"threads_started":2,"wal-append.queue_time_us":210246}
I20260709 13:04:06.316592 14839 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:06.375203 14816 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:06.344509 14681 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:06.379421  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:06.389153  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:06.389930  9323 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.364723 14756 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.371421 14442 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:04:06.370035 14347 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:06.322436 14505 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.301563 14566 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:06.372169 14267 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:04:06.409725 14868 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.270s	user 0.001s	sys 0.001s
I20260709 13:04:06.413623 14728 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.414470  9323 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:06.389683 14681 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.417039 14681 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.422561  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.423475 14761 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:06.375408 14688 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.392544 14827 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:06.433811 14827 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:06.434082 14665 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:06.434531 14827 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.436715 14656 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:06.438104 14871 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 13:04:06.398427 14816 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:04:06.440205 14866 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
W20260709 13:04:06.441977 14763 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.710s	user 0.009s	sys 0.005s
I20260709 13:04:06.442554  9325 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:06.443392 14759 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:06.446110 14869 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
I20260709 13:04:06.446576 14812 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.448477 14505 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.454188 14765 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:06.460958  8655 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:06.467823  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:06.469549  9326 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:06.486110  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:06.487110  9323 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:06.505223  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:06.506304  9323 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:06.498160  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.508692  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 1339) took 1130 ms. Trace:
I20260709 13:04:06.511338  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:06.512269  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.402252 14347 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.605443 14733 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.624570  9327 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:06.509229  9322 rpcz_store.cc:276] 0709 13:04:05.351680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.351882 (+   202us) service_pool.cc:224] Handling call
0709 13:04:06.481702 (+1129820us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b5400552654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:06.482021 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898755840}
I20260709 13:04:06.628595  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:06.559448  9329 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.014s	sys 0.042s
W20260709 13:04:06.679559  9329 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.020s	sys 0.048s
W20260709 13:04:06.640065  8653 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.000s
I20260709 13:04:06.662771 14759 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:06.665544 14756 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.666210  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:04:06.670117 14874 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.294s	user 0.001s	sys 0.000s
I20260709 13:04:06.589246 14347 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:06.633999  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.676035  8724 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:06.677254  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.626087 14769 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.630184 14812 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.622252  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.680392  8653 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:06.688102  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:06.693989  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:06.699997 14347 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:06.701511 14759 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.702528 14756 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:06.703192 14686 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.899s	user 0.009s	sys 0.004s
W20260709 13:04:06.703455 14508 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.000s
I20260709 13:04:06.706163  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.729663 14508 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.732427 14508 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.711561  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.679608  9330 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:06.623857 14823 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:06.634285 14732 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.919s	user 0.013s	sys 0.000s
I20260709 13:04:06.724638  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:06.634884 14267 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:06.728693 14759 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.624310 14566 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:06.713383  8698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.025s	sys 0.040s
W20260709 13:04:06.807693  8698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.025s	sys 0.040s
I20260709 13:04:06.648346 14505 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:04:06.808374  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 851425us
I20260709 13:04:06.726068 14756 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:06.815958  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.789585  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:06.789932  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:06.790223  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.687515 14800 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.812368  8730 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:06.813194  8724 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:06.825289  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:06.766347 14686 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.836704  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:06.831475  8733 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:06.716813 14812 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.821410 14839 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.833737 14859 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:06.825285 14759 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:06.877545 14873 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:06.878070 14759 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.878553 14759 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.828962 14566 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.828030 14505 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:06.884817 14442 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16.
I20260709 13:04:06.884491 14505 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.885651 14442 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:06.886140 14442 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.888098 14442 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.865976  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:06.828680 14756 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:06.758172 14519 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:06.874650  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1336) took 1034 ms. Trace:
I20260709 13:04:06.815925 14267 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.852067 14801 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:06.901232  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.765862 14769 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.895412  9329 rpcz_store.cc:276] 0709 13:04:05.840859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.841011 (+   152us) service_pool.cc:224] Handling call
0709 13:04:05.856032 (+ 15021us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:05.856293 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.856344 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:05.923831 (+ 67487us) log.cc:844] Serialized 26 byte log entry
0709 13:04:06.680092 (+756261us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:06.680135 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.680200 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.874419 (+194219us) raft_consensus.cc:1685] finished
0709 13:04:06.874441 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.874576 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":719078,"threads_started":1,"wal-append.queue_time_us":921801}
I20260709 13:04:06.816095 14732 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.866647 14765 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:06.877622 14733 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.894994 14756 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:06.829934 14800 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.932909  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.895028 14812 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.915602  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.935525  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:06.921969  8730 raft_consensus.cc:2410] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 13:04:06.851887 14686 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:06.935170  8727 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:06.870891 14748 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:06.791168 14823 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:06.892445 14734 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:06.889191 14839 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:06.932191 14756 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:06.944339 14665 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:06.940234 14812 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:06.945465 14765 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:04:06.950490 14688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.061s	sys 0.080s
W20260709 13:04:06.948711 14761 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.016s	sys 0.015s
W20260709 13:04:06.948763 14728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.011s	sys 0.020s
W20260709 13:04:06.952461 14681 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.535s	user 0.009s	sys 0.002s
I20260709 13:04:06.946399 14859 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:06.967494  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.974613  8728 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:06.937914 14801 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
W20260709 13:04:06.973330 14761 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.016s	sys 0.015s
W20260709 13:04:06.973680 14728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.012s	sys 0.021s
W20260709 13:04:06.970374 14688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.061s	sys 0.081s
I20260709 13:04:06.980436 14681 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:06.989583  8583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.015s	sys 0.007s
W20260709 13:04:06.953361 14057 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.009s	sys 0.023s
W20260709 13:04:06.999166 14057 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.009s	sys 0.023s
W20260709 13:04:06.999620  8583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.016s	sys 0.008s
W20260709 13:04:07.002007  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 313951us
I20260709 13:04:06.972692 14665 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:07.002972  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.012413  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:06.979193 14765 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:07.016746  8728 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.016577  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.023077  8731 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.014545  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.024906  8653 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:04:07.025374  8653 pending_rounds.cc:85] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 11
I20260709 13:04:07.025058  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.025691  8653 pending_rounds.cc:107] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:04:06.991225 14859 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:06.930768 14769 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.026098  8653 raft_consensus.cc:2889] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:06.991505 14761 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:07.024848  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:07.028075  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.025940  8655 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:07.030071  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.030881  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.034256  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.034873  8654 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.038599 14267 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.038386 14761 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:07.040940 14827 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.606s	user 0.006s	sys 0.004s
I20260709 13:04:07.041574  8656 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:07.041805 14665 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:07.031616 14838 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.709s	user 0.007s	sys 0.005s
I20260709 13:04:07.048540  8655 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:07.049252  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.051945  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.052693  8731 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.053543  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.054064  8731 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:07.054793  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.026561 14728 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:07.017601 14812 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.056372  8726 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 13:04:07.059036  8726 pending_rounds.cc:85] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 9
I20260709 13:04:07.058877  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.059432  8726 pending_rounds.cc:107] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 13:04:07.059751  8726 raft_consensus.cc:2889] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:07.056695  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.060336  8731 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:07.031538 14748 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:07.063705  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.064126  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.045449 14838 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.077847  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.072741  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.060338 14267 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:07.064949 14748 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.055747 14728 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.067631 14816 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.047784 14827 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.082219  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.091132  8728 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:07.094108  8724 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.085208  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.069093 14823 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:07.089821  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 575054us
I20260709 13:04:07.089900  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.072001  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.072347 14734 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:07.102399  8726 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:07.086277 14748 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:07.102967  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.099661  8725 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.064548 14812 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.090258 14859 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.105877 14800 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.142335  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.088918 14728 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:07.087502 14519 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:07.143611  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.147728  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.111390  8654 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:07.129310  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.130841  8651 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: [10.9-11.10]
I20260709 13:04:07.149881  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.150465  8651 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.068151 14765 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.154711  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.141330 14663 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.151283  8656 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20260709 13:04:07.142987 14688 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.153353  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.095122 14816 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:07.163607  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.126360 14823 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:07.088347 14267 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.174950  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.175041  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:07.188364  8654 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.175369  8655 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:07.155599  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.189854  8650 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:07.156774  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.190855  8653 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:07.176579  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.172227  8659 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:04:07.176604  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.127265 14881 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:07.201550  8653 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
W20260709 13:04:07.180230  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 132883us
I20260709 13:04:07.180327  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.238610  8651 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:07.181573 14663 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.242098 14663 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.242694 14663 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:07.243211 14663 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.174067 14839 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.177676 14732 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:07.255152  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.187570 14519 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:07.210855 14681 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.245563 14663 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.253921  8654 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:07.247910 14877 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:07.247154 14881 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:07.248905 14734 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:07.258708 14839 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:07.281035  8725 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:07.283244  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.284026  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:07.276655  8623 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.778s (SENT) [suppressed 35 similar messages]
I20260709 13:04:07.285442  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.196813 14823 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.187258 14688 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.249576 14733 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 13:04:07.292205  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.273114 14893 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.295325  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:07.289522 14761 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 13:04:07.298595  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.298804  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:07.292834 14880 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 13:04:07.304236  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.264830 14508 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 13:04:07.299412 14839 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.309317  9322 raft_consensus.cc:2410] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 13:04:07.309427 14769 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.312028 14812 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:07.312175 14870 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.065s	user 0.003s	sys 0.000s
W20260709 13:04:07.312989 14883 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 13:04:07.287997 14827 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:07.321086  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.316735 14681 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:07.333161  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.333545  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:07.336935  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:07.337935  9324 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.317664 14838 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:07.332428 14879 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260709 13:04:07.342563  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.342641  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:07.347225  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:07.348773  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 72153.5us
I20260709 13:04:07.349095  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.351171  8732 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:07.352056 14882 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
W20260709 13:04:07.353976 14888 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:07.353907 14759 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.338796 14681 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.356824 14681 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.357653 14267 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.357947  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:07.358892 14892 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:07.359630  8732 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:04:07.343422 14881 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.362202  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.363273  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:07.340921 14734 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.365468 14759 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.366991  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:07.367803  9330 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:07.369431  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1337) took 1225 ms. Trace:
I20260709 13:04:07.305583 14893 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.369796  9328 rpcz_store.cc:276] 0709 13:04:06.143594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.143712 (+   118us) service_pool.cc:224] Handling call
0709 13:04:06.170130 (+ 26418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.294749 (+124619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.369180 (+1074431us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400551c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:07.369344 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":273876736}
I20260709 13:04:07.371676  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.372241  9327 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:07.372737  9324 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:07.388602  8651 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:07.401108 14839 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.404619  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2538) took 1782 ms. Trace:
I20260709 13:04:07.405045  8576 rpcz_store.cc:276] 0709 13:04:05.621990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.829813 (+207823us) service_pool.cc:224] Handling call
0709 13:04:05.830155 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:05.830966 (+   811us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.831003 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:05.831515 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 13:04:05.960008 (+128493us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:07.404028 (+1444020us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.404287 (+   259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.404366 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:07.404408 (+    42us) raft_consensus.cc:1685] finished
0709 13:04:07.404422 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.404560 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1443682,"raft.queue_time_us":195,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":212,"thread_start_us":127670,"threads_started":1,"wal-append.queue_time_us":349543}
I20260709 13:04:07.407047  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.407863  8576 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:07.408810  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.410754  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.409343  8576 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:07.411814  9324 raft_consensus.cc:2368] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
W20260709 13:04:07.413180  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 173772us
I20260709 13:04:07.414332  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:07.415781  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:07.424151  8625 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:07.425623 14898 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:07.425530  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.426994 14812 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:07.436556 14898 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:07.422230 14894 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:07.423269 14881 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:07.438807 14907 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20260709 13:04:07.441574  8625 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.442317 14899 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:07.447861  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.447902 14899 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:07.464380 14839 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.464826  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.465052  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:07.464962 14839 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.464835 14899 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.466480  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.466877 14899 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:07.467751  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:07.468310 14839 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.468688  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.468830  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.465310 14907 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:07.469565 14907 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:07.470064  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.470769  8732 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.470423 14907 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.471557  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:07.471736 14347 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.474673  8731 raft_consensus.cc:2410] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 13:04:07.476148  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.467202  8729 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.485548  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.489902  8576 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:07.497323  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.461761 14688 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.502897  8550 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.511376  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.504829  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1189) took 1099 ms. Trace:
I20260709 13:04:07.447280 14823 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:07.514096  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2872) took 1549 ms (client timeout 1624 ms). Trace:
I20260709 13:04:07.514870  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.515352  8650 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:04:07.517575 14821 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:07.474149 14907 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:07.466931 14765 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:07.467211 14681 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:07.498471 14756 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.001s
I20260709 13:04:07.553700  9324 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
W20260709 13:04:07.537648  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 96145us
I20260709 13:04:07.498442 14839 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:07.554412  8658 rpcz_store.cc:269] 0709 13:04:05.964356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.964477 (+   121us) service_pool.cc:224] Handling call
0709 13:04:07.513860 (+1549383us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b5400139054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:07.514018 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565967232}
I20260709 13:04:07.520439 14663 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:07.562433  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 145100us
I20260709 13:04:07.563848  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1183) took 1889 ms. Trace:
I20260709 13:04:07.498095 14896 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:07.570782  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.585337  8625 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.575115 14905 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:07.575881 14897 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:07.553959  8576 rpcz_store.cc:276] 0709 13:04:06.405853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.408587 (+1002734us) service_pool.cc:224] Handling call
0709 13:04:07.504770 (+ 96183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.582427  8584 rpcz_store.cc:276] 0709 13:04:05.674745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.308393 (+633648us) service_pool.cc:224] Handling call
0709 13:04:06.308713 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.308925 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:07.509666 (+1200741us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.563382 (+ 53716us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.563464 (+    82us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:07.563531 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.563625 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.563774 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1114081,"prepare.queue_time_us":1117524,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"thread_start_us":136247,"threads_started":2,"wal-append.queue_time_us":51748}
I20260709 13:04:07.562939 14898 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:07.588943  9329 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:07.562425 14686 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.595798  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.596868 14898 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.607703 14898 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.602149 14812 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:07.608278 14812 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.608683 14812 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:07.609836  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.610558  8584 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 11.
I20260709 13:04:07.611094  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3134) took 1242 ms. Trace:
I20260709 13:04:07.611416  8584 rpcz_store.cc:276] 0709 13:04:06.368456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.602451 (+1233995us) service_pool.cc:224] Handling call
0709 13:04:07.611001 (+  8550us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.612409  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.613178  8584 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 11: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:07.600589 14881 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 13:04:07.614113 14881 consensus_queue.cc:1243] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 12.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:04:07.571884 14681 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.570668 14765 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.617322 14728 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.618172 14566 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.623422 14728 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.623291  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:07.626852 14876 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.720s	user 0.001s	sys 0.004s
I20260709 13:04:07.627017  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:07.600198 14894 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:07.628119 14894 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.628484 14894 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:07.628707  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1179) took 2001 ms. Trace:
I20260709 13:04:07.629096  8575 rpcz_store.cc:276] 0709 13:04:05.627631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.836796 (+209165us) service_pool.cc:224] Handling call
0709 13:04:05.857276 (+ 20480us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:05.881288 (+ 24012us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.881354 (+    66us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:05.921530 (+ 40176us) log.cc:844] Serialized 52 byte log entry
0709 13:04:06.466699 (+545169us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:06.466745 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.466819 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:07.626294 (+1159475us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:07.628401 (+  2107us) raft_consensus.cc:1685] finished
0709 13:04:07.628424 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.628646 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":915,"raft.queue_time_us":1996,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":116,"spinlock_wait_cycles":54681984,"thread_start_us":433356,"threads_started":1,"wal-append.queue_time_us":945896}
I20260709 13:04:07.630985  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.632187  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3132) took 1263 ms. Trace:
I20260709 13:04:07.632493  8575 rpcz_store.cc:276] 0709 13:04:06.367972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.630711 (+1262739us) service_pool.cc:224] Handling call
0709 13:04:07.631630 (+   919us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.633347  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.633533 14885 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:07.600042 14759 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:07.633998 14885 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:07.634227 14759 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.634606 14759 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:07.635327  8581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1175) took 2009 ms (client timeout 1982 ms). Trace:
I20260709 13:04:07.634999 14885 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:07.635699  8581 rpcz_store.cc:269] 0709 13:04:05.626330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.754813 (+128483us) service_pool.cc:224] Handling call
0709 13:04:07.635142 (+1880329us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b54001685d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:07.635261 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1301384576}
I20260709 13:04:07.636075 14885 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.636754  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.581032 14756 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.675580  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.634723  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2745) took 1662 ms. Trace:
I20260709 13:04:07.647495  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2540) took 1979 ms. Trace:
I20260709 13:04:07.678253  8583 rpcz_store.cc:276] 0709 13:04:05.667775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.273655 (+605880us) service_pool.cc:224] Handling call
0709 13:04:06.293068 (+ 19413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.293565 (+   497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.293609 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:06.324412 (+ 30803us) log.cc:844] Serialized 26 byte log entry
0709 13:04:07.000121 (+675709us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:07.573445 (+573324us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.607236 (+ 33791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.607343 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:07.647195 (+ 39852us) raft_consensus.cc:1685] finished
0709 13:04:07.647219 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.647415 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":452999,"prepare.queue_time_us":426471,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":108986,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":18465,"thread_start_us":762287,"threads_started":3,"wal-append.queue_time_us":723779}
I20260709 13:04:07.680661  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.681293  8583 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:04:07.619396 14681 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.681636 14305 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:07.621161 14765 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.637259 14898 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.620777 14732 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:07.619555  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:07.657184  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2556) took 1180 ms. Trace:
I20260709 13:04:07.690485  8584 rpcz_store.cc:276] 0709 13:04:06.476628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.612119 (+1135491us) service_pool.cc:224] Handling call
0709 13:04:07.657123 (+ 45004us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.676255  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:07.658594 14817 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:07.637316  8581 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:04:07.684244  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2557) took 1207 ms. Trace:
I20260709 13:04:07.693933  8583 rpcz_store.cc:276] 0709 13:04:06.476863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.680349 (+1203486us) service_pool.cc:224] Handling call
0709 13:04:07.684186 (+  3837us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.662022  8725 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:07.678164  8727 rpcz_store.cc:276] 0709 13:04:05.972235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.972466 (+   231us) service_pool.cc:224] Handling call
0709 13:04:07.634516 (+1662050us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54002044d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:07.634643 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066834688}
I20260709 13:04:07.686942  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:07.696270  8729 raft_consensus.cc:2410] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 13:04:07.688572  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2531) took 2068 ms. Trace:
I20260709 13:04:07.689422  8652 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:07.583442 14821 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.667178  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2552) took 1191 ms. Trace:
I20260709 13:04:07.699445  8575 rpcz_store.cc:276] 0709 13:04:06.475423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.633082 (+1157659us) service_pool.cc:224] Handling call
0709 13:04:07.667082 (+ 34000us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:07.700171  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 80685.7us
I20260709 13:04:07.696955  8578 rpcz_store.cc:276] 0709 13:04:05.620038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.786005 (+165967us) service_pool.cc:224] Handling call
0709 13:04:05.834680 (+ 48675us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:05.839521 (+  4841us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.839557 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:05.851636 (+ 12079us) log.cc:844] Serialized 26 byte log entry
0709 13:04:06.057559 (+205923us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:07.311563 (+1254004us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:07.561002 (+249439us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.684546 (+123544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.684641 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:07.684684 (+    43us) raft_consensus.cc:1685] finished
0709 13:04:07.688370 (+  3686us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.688518 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1394221,"prepare.queue_time_us":1368351,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":54365,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":59561,"spinlock_wait_cycles":13076224,"thread_start_us":475261,"threads_started":3,"wal-append.queue_time_us":332235}
I20260709 13:04:07.685647 14305 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:07.659446 14859 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.685442 14769 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:07.710089  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.711058  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1182) took 2043 ms. Trace:
I20260709 13:04:07.712471  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.714339  8583 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:07.715345  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1172) took 2118 ms. Trace:
I20260709 13:04:07.715416  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.659221 14896 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 13:04:07.667672 14748 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.690188 14732 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:07.716307  8583 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:07.717805  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.724742  8575 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:07.717250  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.709695 14765 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:07.701761 14505 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:07.722564  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2528) took 2101 ms. Trace:
W20260709 13:04:07.717995  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 206703us
W20260709 13:04:07.721359 14920 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 13:04:07.711755 14817 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:07.734253  8581 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:07.734277  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.735078  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2555) took 1258 ms. Trace:
I20260709 13:04:07.735509  8581 rpcz_store.cc:276] 0709 13:04:06.476323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.636538 (+1160215us) service_pool.cc:224] Handling call
0709 13:04:07.735015 (+ 98477us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.735937  8654 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:04:07.736202  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.736917  8575 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:07.736698  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.716689  8582 rpcz_store.cc:276] 0709 13:04:05.667442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.224298 (+556856us) service_pool.cc:224] Handling call
0709 13:04:06.224675 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.224931 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:07.511534 (+1286603us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.710551 (+199017us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.710634 (+    83us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:07.710700 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.710792 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.710984 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1198298,"prepare.queue_time_us":1201312,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":72701,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":17029,"thread_start_us":287697,"threads_started":3,"wal-append.queue_time_us":59873}
I20260709 13:04:07.719834 14347 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:07.742385  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2551) took 1267 ms. Trace:
I20260709 13:04:07.742864  8581 rpcz_store.cc:276] 0709 13:04:06.475041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.736477 (+1261436us) service_pool.cc:224] Handling call
0709 13:04:07.742319 (+  5842us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.744395  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.745112  8578 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:04:07.745188  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.737998  8580 rpcz_store.cc:276] 0709 13:04:05.618791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.618929 (+   138us) service_pool.cc:224] Handling call
0709 13:04:05.646668 (+ 27739us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:05.655827 (+  9159us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:04:05.761727 (+105900us) log.cc:844] Serialized 14 byte log entry
0709 13:04:06.156484 (+394757us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:06.184799 (+ 28315us) log.cc:844] Serialized 52 byte log entry
0709 13:04:06.184955 (+   156us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:07.509888 (+1324933us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.718031 (+208143us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.720245 (+  2214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.720324 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:07.720384 (+    60us) raft_consensus.cc:1685] finished
0709 13:04:07.720398 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.720536 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1339766,"prepare.queue_time_us":1304364,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":100,"raft.queue_time_us":138735,"raft.run_cpu_time_us":682,"raft.run_wall_time_us":85572,"thread_start_us":647282,"threads_started":5,"wal-append.queue_time_us":581977,"wal-append.run_cpu_time_us":2268,"wal-append.run_wall_time_us":1044419}
I20260709 13:04:07.745872  8581 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:04:07.733234  8657 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:07.716933 14859 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.730165 14505 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:07.759292  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.715874  8577 rpcz_store.cc:276] 0709 13:04:05.596855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.597071 (+   216us) service_pool.cc:224] Handling call
0709 13:04:05.597521 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.597808 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:05.598212 (+   404us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.402326 (+1804114us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18004afa40. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:04:07.402556 (+   230us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.714921 (+312365us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.714992 (+    71us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:07.715057 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.715131 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.715250 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9434,"prepare.queue_time_us":1651179,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":1045,"raft.queue_time_us":82219,"raft.run_cpu_time_us":417,"raft.run_wall_time_us":76916,"spinlock_wait_cycles":24027008,"thread_start_us":475710,"threads_started":3,"wal-append.queue_time_us":433666,"wal-append.run_cpu_time_us":2352,"wal-append.run_wall_time_us":1359007}
W20260709 13:04:07.752501  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.719277 14769 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.737828 14885 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:07.756850  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.758312  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.750237  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.792189  8581 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:07.792246  8578 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:07.792971  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.793175  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.748595 14894 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.743549 14765 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:07.766654  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.770057 14756 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.770916 14748 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.770994  8580 raft_consensus.cc:1240] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:07.774295 14898 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.764716 14885 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.778425 14859 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.784075 14816 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:07.788177  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:07.791334  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 105193us
I20260709 13:04:07.762301 14505 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.761274  8583 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:07.830943  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.794580  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:07.795161  8581 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:07.795260  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.797700  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2545) took 2112 ms. Trace:
I20260709 13:04:07.808605 14839 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:07.796439 14896 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:07.840215  9325 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1339) took 1535 ms (client timeout 1993 ms). Trace:
W20260709 13:04:07.840621  9325 rpcz_store.cc:269] 0709 13:04:06.305006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.305158 (+   152us) service_pool.cc:224] Handling call
0709 13:04:07.840024 (+1534866us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400702bd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:07.840157 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":313632128}
I20260709 13:04:07.842944  8728 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.845610 14929 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:07.795435 14817 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.754791 14732 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.847044 14732 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:07.741534 14305 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.849772 14305 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.796104 14924 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:07.810077 14812 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.814467  8575 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:07.849792  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.854813  8581 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:07.855171  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1185) took 2093 ms. Trace:
W20260709 13:04:07.817727  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.855499 14812 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.855893  8581 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:07.856501  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3119) took 2078 ms. Trace:
I20260709 13:04:07.857097  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.862149 14800 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:07.826534 14770 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.864336 14347 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:07.864744 14859 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:07.865764 14841 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:07.866146 14841 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:07.866266 14894 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.867074 14841 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.827828 14821 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.828845 14893 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.795877 14663 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:07.868729 14663 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.856802  8581 rpcz_store.cc:276] 0709 13:04:05.777727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.855385 (+2077658us) service_pool.cc:224] Handling call
0709 13:04:07.855730 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.856268 (+   538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.856451 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.870826 14663 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:07.870889  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1186) took 2050 ms. Trace:
I20260709 13:04:07.855552  8575 rpcz_store.cc:276] 0709 13:04:05.761915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.854446 (+2092531us) service_pool.cc:224] Handling call
0709 13:04:07.854719 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.854878 (+   159us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.854902 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.854950 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:07.854982 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.855029 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.855120 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.834609  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1126) took 1207 ms. Trace:
I20260709 13:04:07.871519  8581 rpcz_store.cc:276] 0709 13:04:05.820434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.870037 (+2049603us) service_pool.cc:224] Handling call
0709 13:04:07.870355 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.870538 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.870571 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.870619 (+    48us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:07.870665 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.870722 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.870839 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.873454  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:07.874135  8575 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:07.811578  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3116) took 2092 ms. Trace:
I20260709 13:04:07.867292 14765 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:07.886696  8548 leader_election.cc:400] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:07.887092  8548 leader_election.cc:403] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:07.835668  8577 rpcz_store.cc:276] 0709 13:04:05.685209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.763265 (+2078056us) service_pool.cc:224] Handling call
0709 13:04:07.763558 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.763756 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:07.764957 (+  1201us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.797335 (+ 32378us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.797383 (+    48us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:07.797465 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.797525 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.797632 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":168,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":89,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":31748,"threads_started":1,"wal-append.queue_time_us":32205}
I20260709 13:04:07.873104 14893 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:07.873612  9331 rpcz_store.cc:276] 0709 13:04:06.627381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.627818 (+   437us) service_pool.cc:224] Handling call
0709 13:04:06.738827 (+111009us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.739084 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.739128 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:06.739193 (+    65us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:06.739244 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.739326 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.834460 (+1095134us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b5400542750. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:07.834559 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":114126592}
I20260709 13:04:07.874503  8581 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 13:04:07.890803  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3122) took 2020 ms. Trace:
I20260709 13:04:07.891541  8581 rpcz_store.cc:276] 0709 13:04:05.871196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.873918 (+2002722us) service_pool.cc:224] Handling call
0709 13:04:07.874231 (+   313us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:07.890441 (+ 16210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.890738 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.877084  8584 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:07.877281  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:07.878285  8653 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:07.878118  8580 rpcz_store.cc:276] 0709 13:04:05.719955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.763918 (+2043963us) service_pool.cc:224] Handling call
0709 13:04:07.770815 (+  6897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.811229 (+ 40414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.811431 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.800834 14907 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.880081  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.837363  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.896013  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1187) took 2075 ms. Trace:
W20260709 13:04:07.901073  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 63761.5us
I20260709 13:04:07.902125  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.838008 14763 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:07.906479 14763 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.906857 14763 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:07.889449  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:07.899358  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.910162  8653 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:07.903199  8581 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:04:07.900314 14907 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.904373  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:07.916096  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:07.894963 14663 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:07.916545  8650 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:07.904901 14926 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:04:07.900493 14896 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 13:04:07.905383 14839 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:07.921365 14925 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:07.921881 14925 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:07.922235 14925 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:07.918141 14927 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:07.923858 14927 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.924211 14927 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:07.914798  8578 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:07.925701  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.926317  8658 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:07.926666  8578 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:04:07.927300  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3129) took 1677 ms. Trace:
I20260709 13:04:07.927570  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.930238 14907 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.928308  8653 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:07.932906  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.927594  8578 rpcz_store.cc:276] 0709 13:04:06.250469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.926161 (+1675692us) service_pool.cc:224] Handling call
0709 13:04:07.926469 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:07.927063 (+   594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.927250 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.933940  8577 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 17. Ops: [15.10-15.10]
I20260709 13:04:07.906039 14929 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:07.918237  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.918329  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3124) took 1886 ms. Trace:
I20260709 13:04:07.916273  8656 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:04:07.935382  8581 rpcz_store.cc:276] 0709 13:04:06.031837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.898877 (+1867040us) service_pool.cc:224] Handling call
0709 13:04:07.899177 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:07.911573 (+ 12396us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.917940 (+  6367us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:07.934140  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.941640  8581 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:04:07.943467  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:07.913883 14656 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.942261  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3131) took 1574 ms. Trace:
I20260709 13:04:07.947757  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:07.947721  8581 rpcz_store.cc:276] 0709 13:04:06.367662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.941020 (+1573358us) service_pool.cc:224] Handling call
0709 13:04:07.941308 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.942044 (+   736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.942220 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.948220  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3130) took 1676 ms. Trace:
W20260709 13:04:07.947746  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:07.946617  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48139.1us
I20260709 13:04:07.948827  8578 rpcz_store.cc:276] 0709 13:04:06.271376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.934593 (+1663217us) service_pool.cc:224] Handling call
0709 13:04:07.947756 (+ 13163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.947912 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.947936 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.947977 (+    41us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:07.948007 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.948054 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.948149 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.917006 14885 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:07.959152  8575 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:07.957119  8650 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:07.931516 14841 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.889022 14800 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 13:04:07.925022 14763 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.965893  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:07.966775  8656 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:07.968613  8578 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
I20260709 13:04:07.959079  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:07.901216 14893 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.970268  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:07.971176  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3133) took 1603 ms. Trace:
I20260709 13:04:07.974012  8653 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:07.973971  8578 rpcz_store.cc:276] 0709 13:04:06.368224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.950161 (+1581937us) service_pool.cc:224] Handling call
0709 13:04:07.968400 (+ 18239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.970964 (+  2564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.971131 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.971365  8650 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:07.914083 14759 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:07.971653  8575 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:04:07.950841 14770 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.973387  8581 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:07.950403 14924 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:07.976240  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1190) took 1522 ms. Trace:
I20260709 13:04:07.913699 14688 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 13:04:07.960341 14663 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:07.962227 14929 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:07.981966  8575 rpcz_store.cc:276] 0709 13:04:06.454418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.963297 (+1508879us) service_pool.cc:224] Handling call
0709 13:04:07.964878 (+  1581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.976001 (+ 11123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.976189 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:07.917775 14505 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:07.988320  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.988425  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1180) took 1152 ms. Trace:
I20260709 13:04:07.989387  8724 rpcz_store.cc:276] 0709 13:04:06.836075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.836372 (+   297us) service_pool.cc:224] Handling call
0709 13:04:07.988167 (+1151795us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54000ea454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:07.988352 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584514944}
I20260709 13:04:07.992254  8655 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:07.995790  8581 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:07.996618  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2554) took 1520 ms. Trace:
I20260709 13:04:07.996994  8581 rpcz_store.cc:276] 0709 13:04:06.476016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.978782 (+1502766us) service_pool.cc:224] Handling call
0709 13:04:07.992336 (+ 13554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.996270 (+  3934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.996546 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:08.001777  8578 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:04:08.002702  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2550) took 1528 ms. Trace:
I20260709 13:04:08.003119  8578 rpcz_store.cc:276] 0709 13:04:06.474599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.975077 (+1500478us) service_pool.cc:224] Handling call
0709 13:04:07.983783 (+  8706us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:08.002333 (+ 18550us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.002623 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:08.022639  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43598.4us
I20260709 13:04:08.032567  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3135) took 1015 ms. Trace:
I20260709 13:04:08.033042  8584 rpcz_store.cc:276] 0709 13:04:07.016988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.693738 (+676750us) service_pool.cc:224] Handling call
0709 13:04:08.032329 (+338591us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b540030be54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:08.032483 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287978880}
I20260709 13:04:08.034523  8584 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:08.036399  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1184) took 2274 ms. Trace:
I20260709 13:04:08.036818  8583 rpcz_store.cc:276] 0709 13:04:05.761630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.831869 (+2070239us) service_pool.cc:224] Handling call
0709 13:04:07.895729 (+ 63860us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:07.895918 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.895939 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:07.919497 (+ 23558us) log.cc:844] Serialized 26 byte log entry
0709 13:04:08.034219 (+114722us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:08.034262 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.034337 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:08.036194 (+  1857us) raft_consensus.cc:1685] finished
0709 13:04:08.036214 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.036348 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":58349,"threads_started":1,"wal-append.queue_time_us":70702}
I20260709 13:04:08.037643  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.039906  8575 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:08.053956  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:07.993052 14663 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.061508  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.059284  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.042081 14817 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:08.062667  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.062915  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.063009  9327 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:08.062515  9324 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:08.081225 14763 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.092612  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1181) took 2430 ms. Trace:
I20260709 13:04:08.061954 14948 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:08.094004 14948 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.094440 14948 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:08.097618 14823 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:08.097770 14947 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:08.098373 14947 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:08.099478 14947 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.102622 14898 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.103178 14898 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.106020 14663 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.111312  8576 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:04:08.089177  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2878) took 1060 ms. Trace:
I20260709 13:04:08.114084  8657 rpcz_store.cc:276] 0709 13:04:07.029173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.029341 (+   168us) service_pool.cc:224] Handling call
0709 13:04:08.088948 (+1059607us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b54000591d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:08.089108 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1571003264}
I20260709 13:04:08.125074  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.125968  8576 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:08.127599  8576 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 13:04:08.093204  8579 rpcz_store.cc:276] 0709 13:04:05.662523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.838866 (+176343us) service_pool.cc:224] Handling call
0709 13:04:05.854557 (+ 15691us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:05.854807 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:07.403514 (+1548707us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.518625 (+115111us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b18003b3f20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a
0709 13:04:08.091854 (+573229us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18003b3f20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287
0709 13:04:08.092113 (+   259us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:08.092185 (+    72us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:08.092259 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.092371 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.092532 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1568553,"prepare.queue_time_us":1565176,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":44402,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7238,"spinlock_wait_cycles":36855040,"thread_start_us":85746,"threads_started":2,"wal-append.queue_time_us":109966}
I20260709 13:04:08.123639 14898 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.064980 14817 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:08.079949 14927 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.165359 14927 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.169689  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.171101  9329 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:08.173853 14656 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:08.176359  9329 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:08.180495  9329 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:04:08.183444  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.184263  9329 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:08.188839 14817 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.190003 14817 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.193897  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.194314  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.194672  9324 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.196138  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.197862  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.197885  8578 raft_consensus.cc:1240] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:08.200542  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.201196  9328 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:08.203667  9324 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:08.214723  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:08.227056  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.165863 14769 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.228821  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.229789  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:08.230046  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:08.230688  8657 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:08.166429 14925 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.233132  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.235639  8623 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:08.235982  8650 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:08.231817  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.236670  8580 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:04:08.233568  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.238869  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:08.240693  9329 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:08.242553  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.241139 14898 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:08.261744 14756 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:08.277645  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:08.255666  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3123) took 2324 ms. Trace:
I20260709 13:04:08.278055  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.278488  8577 rpcz_store.cc:276] 0709 13:04:05.931257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.896568 (+1965311us) service_pool.cc:224] Handling call
0709 13:04:07.933701 (+ 37133us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:07.934547 (+   846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.255381 (+320834us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b54002ca1d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:08.255574 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84462464}
I20260709 13:04:08.278908  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.267436  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.279831  8656 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:08.268191  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.268422  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.265147  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.263415 14505 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.269740 14893 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:08.271095  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 966786us
I20260709 13:04:07.901436  8582 rpcz_store.cc:276] 0709 13:04:05.820738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.870417 (+2049679us) service_pool.cc:224] Handling call
0709 13:04:07.872227 (+  1810us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.872409 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:07.872685 (+   276us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.895700 (+ 23015us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:07.895749 (+    49us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:07.895789 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.895846 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:07.895962 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":631,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":79,"thread_start_us":22619,"threads_started":1,"wal-append.queue_time_us":20858}
I20260709 13:04:08.272209 14873 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:08.272256 14733 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:04:08.292484  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.280738  8657 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:08.280786  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.282599  9322 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:08.293938  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.275401 14812 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:08.280126 14770 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:08.300271  9297 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.283505 14926 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:08.303027  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.300197  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.242457 14656 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:08.286387  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:08.289420 14267 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.258023  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3226) took 1130 ms. Trace:
I20260709 13:04:08.326458  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.326823  8727 raft_consensus.cc:1217] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Deduplicated request from leader. Original: 8.7->[10.8-17.10]   Dedup: 10.8->[15.9-17.10]
I20260709 13:04:08.281046  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:08.296922  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:08.328083  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.326033  8652 rpcz_store.cc:276] 0709 13:04:07.128103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.149192 (+ 21089us) service_pool.cc:224] Handling call
0709 13:04:07.149453 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.697169 (+547716us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.257714 (+560545us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b5400058cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:08.257930 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241229696}
I20260709 13:04:08.292212 14305 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.298094  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.292812 14733 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:08.296233  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.284435  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:08.279841 14907 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:08.305137  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.256925 14947 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:08.305840  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.306806  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.345484  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.308046  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.309201  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:08.309725  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.310518  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.347179  8731 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:08.326390  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:08.327525  8727 pending_rounds.cc:85] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:04:08.335667  8657 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:08.291702  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:08.295075  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.285310 14924 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 13:04:08.337539  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:08.302644 14838 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.281750 14761 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.028s
W20260709 13:04:08.343062  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46901.4us
I20260709 13:04:08.347810  8727 pending_rounds.cc:107] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 13:04:08.280340 14885 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 13:04:08.350682  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.360862  8727 raft_consensus.cc:2889] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:08.337949  8728 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
W20260709 13:04:08.353288  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61678.5us
I20260709 13:04:08.354046  8724 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:08.302959 14899 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:08.362548  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.362890 14899 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.363320 14899 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:08.363451  8724 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:08.364470  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.300557 14816 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:08.353543  8697 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:08.302798 14519 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:08.355690  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49938.3us
I20260709 13:04:08.351759  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.368697  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.295379 14893 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:04:08.298100 14841 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:04:08.333561 14505 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:04:08.390746  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 106374us
I20260709 13:04:08.391631  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.395215  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2561) took 1079 ms. Trace:
I20260709 13:04:08.395429  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.395674  8581 rpcz_store.cc:276] 0709 13:04:07.315727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.997962 (+682235us) service_pool.cc:224] Handling call
0709 13:04:07.999188 (+  1226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.999465 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:08.028469 (+ 29004us) log.cc:844] Serialized 14 byte log entry
0709 13:04:08.394757 (+366288us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:08.394836 (+    79us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:08.394909 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.394998 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.395153 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38047,"prepare.queue_time_us":29938,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":9116,"thread_start_us":327786,"threads_started":1,"wal-append.queue_time_us":328225}
I20260709 13:04:08.374330  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3125) took 2307 ms. Trace:
I20260709 13:04:08.337591 14732 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.298730 14950 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.078s
I20260709 13:04:08.378911 14761 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:08.379341 14885 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:08.355162 14948 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.367064 14859 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:08.380749 14519 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.381886  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.444368  8733 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.362573  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.348596 14907 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:08.386435 14925 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.398365 14769 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:08.328246 14940 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:08.303092 14294 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 13:04:08.331338 14884 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:08.398749  8580 rpcz_store.cc:276] 0709 13:04:06.066631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.894138 (+1827507us) service_pool.cc:224] Handling call
0709 13:04:07.894542 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:08.237309 (+342767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.373949 (+136640us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54000f91d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:08.374246 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":786075008}
I20260709 13:04:08.435036 14924 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:08.365703 14816 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.399669  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2944) took 1294 ms. Trace:
I20260709 13:04:08.338698 14688 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:08.349852 14733 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:08.340257 14812 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:08.417922  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:08.354326 14902 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.432602  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.351243 14656 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:08.366578 14821 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:04:08.390286 14955 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:04:08.416083 14810 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:08.367379 14873 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:08.440827  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 159835us
I20260709 13:04:08.445284  8728 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:08.337265 14267 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.496889  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.342667 14305 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.497535  8728 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:08.499099 14305 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:08.386909 14945 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:08.420925 14505 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:08.471453 14812 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.501689 14812 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.501541 14505 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.372923 14838 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.504285 14838 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.436935 14761 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:08.504925 14761 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.505903 14873 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.465688  8731 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:08.460822 14952 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 13:04:08.510270  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.461663 14827 consensus_queue.cc:1059] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 13:04:08.458415  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:08.511559  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.512203  8731 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:08.432798 14732 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.512481 14519 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.447036 14907 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:08.482023  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.492106 14821 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.445796 14906 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:08.488615  8659 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:08.449317 14769 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:08.461599  8726 rpcz_store.cc:276] 0709 13:04:07.106983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.153882 (+ 46899us) service_pool.cc:224] Handling call
0709 13:04:07.712607 (+558725us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:07.712818 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:07.795510 (+ 82692us) log.cc:844] Serialized 14 byte log entry
0709 13:04:08.290068 (+494558us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:08.290132 (+    64us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:08.290183 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.290249 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.399443 (+109194us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54004218d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:08.399593 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":296978,"prepare.queue_time_us":158775,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":285169,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":5804,"spinlock_wait_cycles":1241117440,"thread_start_us":272823,"threads_started":2,"wal-append.queue_time_us":349204}
W20260709 13:04:08.489132  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 71255.9us
I20260709 13:04:08.520329  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.486697  8724 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:08.495599  8623 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.521682  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:08.510777 14267 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:08.449035 14885 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:08.513854 14821 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:08.494944 14810 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.541582 14810 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:08.543964 14733 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.545060  8732 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.463291  8652 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:04:08.545836 14884 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:08.546104 14885 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.546868 14884 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:08.547565 14884 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:08.550679 14948 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.555219  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.556032  8726 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
W20260709 13:04:08.512286  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 53993us
I20260709 13:04:08.513110  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.454638 14877 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:08.529728  8728 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:08.522449 14906 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:08.532233  8659 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:04:08.489142 14957 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.536028 14812 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.493276 14656 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.580040 14960 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:08.580439 14656 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.492867 14902 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.494971 14924 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:08.538271  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.538486  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.539088 14761 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.525597 14508 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:08.540681 14816 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:08.558226  8625 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:08.568178 14906 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:08.562839  8732 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:08.571398 14877 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.572180 14957 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:08.584781  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 289740us
I20260709 13:04:08.586089  9329 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:08.553987 14733 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.580878 14960 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:08.590392  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.592437  8725 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.585240 14924 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.600430  8653 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:08.625890  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.594604 14899 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:08.589684  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69431.4us
I20260709 13:04:08.548738 14884 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.588523 14925 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:08.606676  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 242116us
W20260709 13:04:08.608558  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 255117us
I20260709 13:04:08.592079 14770 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:08.668742  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:08.617656 14962 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:08.603152  8724 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:04:08.678529  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.680078  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:08.644565 14924 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:08.681813 14812 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.633665 14732 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.693904  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.687673  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:08.695971  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.700141  9329 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:08.701805  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.702615  9325 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:08.703706 14812 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:08.704279 14812 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:08.704681 14759 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.649793 14816 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:08.705176 14812 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.706557 14759 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.707238 14759 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.633019 14508 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 13:04:08.711306 14963 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:08.711329 14838 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:08.712714 14838 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:08.713312 14838 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:08.714308 14305 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:08.714651 14305 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.714838  8732 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:08.715447 14305 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.716171  8732 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 13:04:08.716565  8732 pending_rounds.cc:85] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:04:08.716279 14821 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.716987  8732 pending_rounds.cc:107] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 13:04:08.685866 14962 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.719633 14962 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.675832 14733 consensus_queue.cc:1059] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:04:08.720991 14733 raft_consensus.cc:3055] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:08.721558 14733 raft_consensus.cc:740] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.605577 14898 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:04:08.672856  8725 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:08.599735 14810 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.722644 14733 consensus_queue.cc:260] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.723945 14733 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:08.692261  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.675298 14925 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.716624 14305 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.676097  8726 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:08.604589 14769 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.668879  8733 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:08.717381  8732 raft_consensus.cc:2889] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:08.748492  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2562) took 1400 ms. Trace:
I20260709 13:04:08.748889  8578 rpcz_store.cc:276] 0709 13:04:07.348309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.004081 (+655772us) service_pool.cc:224] Handling call
0709 13:04:08.039328 (+ 35247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:08.198941 (+159613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.748229 (+549288us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54000b6654. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:08.748418 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364193024}
I20260709 13:04:08.647234  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.670544  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:08.720927 14906 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:08.758725  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 71113.2us
W20260709 13:04:08.759166  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47190us
I20260709 13:04:08.759627  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:08.760478 14959 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:08.761145 14959 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:08.762210 14965 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:08.762844 14965 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:08.763392 14965 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.760090 14906 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.764235 14519 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:08.764690 14519 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:08.755224  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 84761.5us
I20260709 13:04:08.756354 14957 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:08.709306 14924 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:08.728613 14963 consensus_queue.cc:1175] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "453e644bad5d410ab9be85b353e28dc2" responder_term: 15 status { last_received { term: 15 index: 12 } last_committed_idx: 11 last_received_current_leader { term: 15 index: 12 } }
I20260709 13:04:08.747551 14816 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.746170 14960 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.766811 14816 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.767410 14924 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.756815 14770 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:08.767923 14838 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:08.768293 14770 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:08.768707 14770 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.770187 14902 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:08.746284 14810 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.757627 14508 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:08.772217 14508 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.762090 14962 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:08.717304 14877 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:08.773154 14954 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.544s	user 0.000s	sys 0.007s
I20260709 13:04:08.776305  9330 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:08.780524 14519 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.792023 14519 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.792080 14962 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:04:08.794375  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 147245us
W20260709 13:04:08.796026  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 445336us
I20260709 13:04:08.797144  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:08.798655  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:08.801239  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:08.803090  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 244896us
I20260709 13:04:08.804083  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:08.830816  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:08.834858  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:08.835657  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.834690  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.836455  8576 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 13:04:08.836838  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:08.837536  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.838310  8576 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:08.839193  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:08.842602  8651 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:08.795686 14962 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.836511  8577 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:08.845541  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:08.849308  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:08.799712 14732 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:08.851281  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.852025  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:08.852984  8726 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:04:08.853523  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:08.854466  8656 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:08.827244 14898 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:08.847018  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.846844  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.864056  8581 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:08.849061  8732 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:08.867295  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.851392  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.857739 14899 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.858042  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.838488 14885 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:08.858482 14962 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.843230 14894 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:08.843341 14810 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.877761  8656 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:04:08.860265 14838 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.857779  8580 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:08.868188  8583 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:08.852419  8578 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:08.868258  8582 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:08.773316 14508 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.873139  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:08.876266 14821 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.850046 14759 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:08.876363  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:08.846475 14948 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 13:04:08.825743 14964 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:04:08.897632 14964 consensus_queue.cc:1243] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:04:08.867147 14732 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.898939 14732 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:08.899894 14769 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:08.872352 14898 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:08.885764  8651 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:08.847453 14938 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:04:08.846400 14347 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:04:08.891520 14508 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:08.889204 14505 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 13:04:08.874221 14885 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.887457 14970 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.873963 14894 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:08.918780 14894 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.920651 14688 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:08.885449 14838 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.923125 14838 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:08.923959 14442 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:08.924394 14442 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:08.887854 14899 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.882313 14817 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 13:04:08.921169 14688 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.907441 14962 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:04:08.882870  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57211.2us
I20260709 13:04:08.888398 14902 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 13:04:08.915103 14885 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:08.924785 14770 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.946593  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:08.949259 14759 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:08.958906  8576 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:04:08.960335  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:08.963140  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.963999  8576 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:08.975555  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.975800  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.979470  9325 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.981632 14969 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:08.982209 14969 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.982544 14969 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.989054 14810 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:08.990197  8576 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:08.994627 14873 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:08.994889  8578 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:08.995965  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:08.996490  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1046) took 1116 ms. Trace:
I20260709 13:04:08.997278  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:08.998194  9331 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:04:09.002992  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47683.1us
I20260709 13:04:09.006127  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.008774  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.009810 14977 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:09.012094 14977 consensus_queue.cc:1059] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:09.012914 14977 raft_consensus.cc:3055] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:08.935802 14442 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.014384 14442 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.996593  8583 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:09.015439  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:09.016296 14812 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.016767 14812 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.017582 14898 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.018821 14812 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:08.927421 14970 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:08.997104  8655 rpcz_store.cc:276] 0709 13:04:07.879712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.879847 (+   135us) service_pool.cc:224] Handling call
0709 13:04:08.996318 (+1116471us) spinlock_profiling.cc:243] Waited 91.6 ms on lock 0x7b5400410854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:08.996439 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210784128}
I20260709 13:04:09.022224 14970 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:09.022637  8623 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:09.023602  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:09.024215  8651 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:09.024447 14759 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:09.025058 14759 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:09.009878  8582 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:08.950013 14841 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.033804  9325 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:09.030242  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.034886  8623 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:09.031157  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:09.031663  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.037025  8730 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:09.032469  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.037701  8583 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:09.037937  8731 raft_consensus.cc:2410] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 17 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:04:08.934685 14899 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:09.048605  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.055095  8651 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:08.946943 14964 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.086s
I20260709 13:04:09.013307 14977 raft_consensus.cc:740] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:08.941920 14961 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.091s
W20260709 13:04:09.065140  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 226144us
W20260709 13:04:09.074594  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 235111us
I20260709 13:04:09.085850  8696 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:09.077253  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46194.6us
I20260709 13:04:08.951458 14756 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 13:04:08.976843 14925 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.024379 14893 raft_consensus.cc:2764] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:04:09.088459  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:08.950452 14974 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.117s
I20260709 13:04:09.100140  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:09.100821 14816 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:09.105268  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.037272 14656 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 13:04:09.109366  8655 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:09.110355  8651 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 10. Current term is 11. Ops: []
W20260709 13:04:09.174366  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 65203.2us
I20260709 13:04:09.174657  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:09.179929  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:09.222131  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.026044 14759 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:09.180585  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65773us
I20260709 13:04:09.224404  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:09.224617  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:09.180733  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 210824us
I20260709 13:04:09.102451 14967 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:09.108475 14983 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:09.053812 14841 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:09.190181 14973 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 13:04:09.190584 14899 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:09.191390 14770 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:09.196111 14965 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.179579 14873 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:09.182217 14977 consensus_queue.cc:260] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:09.202486 14907 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.689s	user 0.010s	sys 0.002s
I20260709 13:04:09.205624 14894 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.213574  8728 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:09.110668 14985 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:09.115356 14986 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.215287 14305 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 13:04:09.115600 14976 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:04:09.131251  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.225656  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.277103  8651 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:09.226140  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.224826  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:09.253079 14873 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.283092  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.284077  8655 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:09.227584  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:09.248286  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.221627 14812 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.251117  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.252172  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.290740  8652 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:09.247725 14983 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:09.259548 14985 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:09.295192  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.226162 14974 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:04:09.250949 14967 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:09.278842  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54087.7us
I20260709 13:04:09.226675  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.302802  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:09.286237  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:09.303998  8651 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:04:09.286172  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59047.5us
I20260709 13:04:09.227277 14893 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
W20260709 13:04:09.288859 14733 consensus_meta.cc:287] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.547s	user 0.011s	sys 0.003s
I20260709 13:04:09.297163  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:09.318060  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:09.292021 14816 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:09.289978 14976 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:09.243865 14748 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 13:04:09.250730 14898 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:09.312063  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 85477.5us
I20260709 13:04:09.287873 14873 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:09.262915 14907 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.332229 14907 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.332859  8654 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:09.283612 14977 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:09.334931 14947 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:09.335397 14947 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:09.301653  8697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.003s	sys 0.019s
W20260709 13:04:09.335907  8697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.003s	sys 0.019s
W20260709 13:04:09.336294  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 574866us
I20260709 13:04:09.337937  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.297505 14985 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:09.341712 14983 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.342804 14983 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:09.343098 14885 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.344391  8657 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:09.322178 14816 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:09.345257 14816 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:09.346643  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.294368 14812 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.349510 14812 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:09.315670 14838 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:09.350286 14838 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:09.323318 14267 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:09.350663 14838 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:09.350930 14267 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:09.351023  8733 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:04:09.351400 14267 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:09.353236  8733 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:09.306398  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:09.311882 14305 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:09.314565 14519 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:09.309460 14732 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 13:04:09.329149  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:09.322851 14989 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:09.339551  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.368319  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:09.371084  8731 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:09.355926 14894 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.378734  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.340965  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.340631  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.284662 14986 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:09.357570  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51231us
I20260709 13:04:09.359999  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:09.364804  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.327585 14906 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:09.367108  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:09.387776  8578 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:09.370941  8726 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:04:09.340487 14987 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260709 13:04:09.370507  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 52510.4us
I20260709 13:04:09.340528  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.394476  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.381551  8731 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:09.384660  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.380379  8730 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:09.363054 14733 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:09.403177  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.403823  8584 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:09.370883 14770 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:09.403823  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.404924  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.356235 14898 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:09.400584  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.408931  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:09.409296  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.411938  8581 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:09.412576  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.413913  8727 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:09.380473 14991 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:09.415498  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.417690  8550 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.418838  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.419523  8730 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:09.421021  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.421743  8732 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:09.422720  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.423991  8731 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:09.429679  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.431751  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.392098 14906 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:09.435248  9297 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.437325  8731 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:09.402187 14733 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:09.441262  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 127502us
I20260709 13:04:09.433564  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.442680  8578 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:09.438830  8580 raft_consensus.cc:1275] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:09.445181  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.445906  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:09.446723  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:09.401875 14885 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.373059 14873 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.442341  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:09.407081 14980 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.454763  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.407217 14770 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:09.412636 14877 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:09.437829 14950 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:09.443363 14906 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:09.478451  8580 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 13:04:09.479225  8580 pending_rounds.cc:85] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 8
I20260709 13:04:09.481868  8580 pending_rounds.cc:107] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 13:04:09.447372 14733 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:09.451334 14656 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:09.476372 14974 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:09.484171  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:09.460474 14942 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:09.407791 14947 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.438035 14952 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 13:04:09.456599 14965 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.467585 14770 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:09.473292 14893 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:09.495189 15007 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:09.497069 14993 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 13:04:09.400584 14967 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.413594 14976 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.482791  8580 raft_consensus.cc:2889] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:09.523687  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68965.3us
I20260709 13:04:09.524844  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.491969 14950 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:09.492452 14974 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:09.515079 14969 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.000s
I20260709 13:04:09.531983  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.506217 14810 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:09.507355 14873 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:09.535511 14952 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:09.507634 14947 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:09.509239 14926 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.466090 14980 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.541359 14841 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:09.545739 14986 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.531898 14821 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 13:04:09.532618 14974 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:09.569603  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.580889  8730 raft_consensus.cc:2410] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
W20260709 13:04:09.581493  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.520548 15007 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.593925  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.572361 14952 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:09.603377 14827 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:09.603574 14980 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.603885 14827 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:09.604933 14827 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.607054 14827 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:09.597078  8580 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:04:09.599699 14962 consensus_meta.cc:287] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.656s	user 0.009s	sys 0.001s
I20260709 13:04:09.601364 15007 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:09.602083 15004 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 13:04:09.532120  8732 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:09.616643  8577 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:09.616879  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.541877 14969 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.620560  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.628973  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:09.629621  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 164228us
I20260709 13:04:09.617638  8732 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:09.618445  9297 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:09.634716  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:09.636010  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.636642  8732 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:09.640375  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.664412  8732 raft_consensus.cc:2410] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:04:09.648053  8658 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:04:09.570094 14926 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.658574  8655 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:09.608623 14838 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.646868  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:09.669128  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:09.699532  8732 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:09.616216 14952 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.655298 14821 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:09.655669 14505 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:09.686515  8730 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: [13.10-15.11]
I20260709 13:04:09.649417 14898 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:09.683596 15001 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
I20260709 13:04:09.658624 14761 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:09.693125 14688 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.743s	user 0.006s	sys 0.002s
W20260709 13:04:09.710037  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63245.8us
I20260709 13:04:09.674819  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:09.657367 14962 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:04:09.683192 14969 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.696799 15003 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:09.711826 14907 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:09.727041 14508 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.811s	user 0.012s	sys 0.000s
I20260709 13:04:09.746748  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:09.764071  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:09.720732 14821 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:09.813948  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48767.2us
W20260709 13:04:09.824875  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.683357 14985 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.836428 15011 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:09.837108 15011 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.839676 15011 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.831413 14980 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.844319 14980 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.846511 14980 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:09.847864 15007 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.848428 15007 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:09.849949 15022 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 13:04:09.850652 15007 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:09.851850 14838 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:09.802416 14267 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.853191  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.854511  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.855239  8580 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:09.855101 14821 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.731336 14816 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.859314 14816 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:09.859735  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.861660 14942 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:09.738610 14688 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.863201  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.863526 14770 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.864276 14688 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:09.749316 15017 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:09.874800  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59942.8us
I20260709 13:04:09.871520  8580 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:09.881232  8733 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:09.882423  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:09.873103 14969 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:09.854488 14267 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:09.870072  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 195383us
I20260709 13:04:09.897450  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:09.897696 15017 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:09.899755 14761 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:09.901901  8580 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:09.902011 14906 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.903421  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:09.904835  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:09.910674 14952 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:09.894665 15005 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:09.892388 15020 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:09.939899  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:09.938391 15019 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.131s
I20260709 13:04:09.967726 15003 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:04:09.941103  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 523377us
I20260709 13:04:09.940186  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:09.952503  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.968475 15003 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:09.952221  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:09.964965  8581 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:09.970722 14884 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:09.971359 14884 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.972712  8732 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:09.974035 14884 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.975073  8732 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:09.969048 15003 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:09.976910  8732 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:09.970772 14898 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.978441 14898 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:09.979730  8732 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:09.980757 14898 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.977046 14961 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:04:09.983875  8732 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:09.968083 15019 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:09.883658 14770 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:09.985972  8724 raft_consensus.cc:1240] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:09.968972  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.938956 14267 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:09.993659  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.940472 14967 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:09.996743  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.943018 14748 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:09.998385  8578 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:09.999977  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:10.001730  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:10.005178  8732 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:10.007323  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:10.000927  8577 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
W20260709 13:04:09.997112 14733 consensus_meta.cc:287] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.504s	user 0.009s	sys 0.001s
I20260709 13:04:10.019275  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:09.991467 15005 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.023370  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:10.000955 15019 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:10.002158 14770 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:10.044493  8732 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:10.002908 14267 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:10.046312  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:10.018862 14810 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:10.039923 15005 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:10.059362 14926 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.059840 15030 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:10.040705 14961 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 13:04:10.004522 14894 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 13:04:10.061766 14985 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:10.062537 15012 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.228s	user 0.000s	sys 0.005s
I20260709 13:04:10.062440 14894 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:10.063542 14761 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:10.063921 14761 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:10.059895 14267 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.064713 14761 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.066277 14761 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:10.021481 14884 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:04:10.046572 15019 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:10.052337 14907 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:10.020327 14748 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.073557 15003 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.039389 14733 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.081862  8732 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:10.082427  8732 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:10.085319  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.043191 14977 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 13:04:10.090067 15009 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:10.090837 15021 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:10.091308 15021 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:10.091681 15021 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:10.092382 15010 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:04:10.098076 15013 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 13:04:10.102095 15028 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:04:10.056870 14893 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:04:10.112468  8731 raft_consensus.cc:2368] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
I20260709 13:04:10.083369  8732 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.115428  8732 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:10.080809 14907 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.116621 14759 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:10.070480 14770 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 13:04:10.075114 14967 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 13:04:10.126065 15003 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:10.120805 14907 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:10.140269 14974 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.556s	user 0.011s	sys 0.000s
I20260709 13:04:10.143517  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1061) took 1743 ms. Trace:
I20260709 13:04:10.143970  8654 rpcz_store.cc:276] 0709 13:04:08.400175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.514836 (+114661us) service_pool.cc:224] Handling call
0709 13:04:08.515194 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:09.333274 (+818080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.143244 (+809970us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b540018e7d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:10.143449 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1875093888}
I20260709 13:04:10.146682 14899 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:10.147145 14899 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:10.148106 14899 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:10.151186 15034 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 13:04:10.145344  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:10.140800 14974 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:10.170143  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.170229  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:10.137048 14906 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.175319  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:10.182097  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:10.182832 14974 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:10.184098 15037 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 13:04:10.184559  8583 raft_consensus.cc:2393] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:10.186229  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:10.187193  8583 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:10.189790 14748 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.164179 14962 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.168598 14952 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:10.185657 14994 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.868s	user 0.001s	sys 0.004s
I20260709 13:04:10.168498 15017 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:10.183081 14884 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:10.202358  8732 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:10.204792  8654 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:10.212493  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:10.213297  8654 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:04:10.214911  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:10.188264 14899 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:10.199903  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:10.215540  8581 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:10.222903  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:10.224570  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 171989us
I20260709 13:04:10.225991  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:10.230541  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2778) took 1398 ms. Trace:
I20260709 13:04:10.235656  8581 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:10.231083  8725 rpcz_store.cc:276] 0709 13:04:08.832303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.846803 (+ 14500us) service_pool.cc:224] Handling call
0709 13:04:10.230289 (+1383486us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400203fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:10.230469 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":833390336}
I20260709 13:04:10.237162  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:10.214601 14980 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:10.215500 14974 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.240396  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:10.241117  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:10.198133 14907 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.243100  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.243414  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.244681  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.245229  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.206447 14962 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.129611 15019 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.208900 14952 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.247174  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:10.332515  9328 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:10.332772  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 85674.2us
I20260709 13:04:10.333745  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:10.238897 15023 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 13:04:10.327251  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.325802  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.239378 14841 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:10.315960 15035 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:04:10.331686  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.335100  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:10.335453  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.292474  9323 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:10.329694  9325 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:10.326273  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.346935  9322 raft_consensus.cc:2368] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15.
I20260709 13:04:10.335961  8729 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:04:10.338452  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.341822  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1203) took 1388 ms. Trace:
I20260709 13:04:10.349448  8575 rpcz_store.cc:276] 0709 13:04:08.953426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.953546 (+   120us) service_pool.cc:224] Handling call
0709 13:04:08.953915 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:08.954297 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:08.954339 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:08.971109 (+ 16770us) log.cc:844] Serialized 26 byte log entry
0709 13:04:09.216891 (+245782us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:09.216930 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.217001 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.184948 (+967947us) spinlock_profiling.cc:243] Waited 6.24 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:10.333133 (+148185us) raft_consensus.cc:1685] finished
0709 13:04:10.333153 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.341750 (+  8597us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":689,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":14260,"spinlock_wait_cycles":14347008,"thread_start_us":223315,"threads_started":1,"wal-append.queue_time_us":267459}
I20260709 13:04:10.340618  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:10.353873  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:10.354938  8725 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:10.242017 14765 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:10.366179  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.356319  9331 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:10.314173 15020 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.314419 15003 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.245924 14906 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:10.239753 14821 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.371579  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:10.374061  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.383018 15044 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:10.316505 14950 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 13:04:10.240595 14733 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:10.218536 14884 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.390494  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:10.391775  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:10.362512  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:10.245821 14688 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.147s
I20260709 13:04:10.387560  8654 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:10.638496  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 2976) took 1194 ms. Trace:
I20260709 13:04:10.541935  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1190) took 1204 ms. Trace:
I20260709 13:04:10.542201  8656 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
W20260709 13:04:10.608417  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.552008  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.601302  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.654922  8731 rpcz_store.cc:276] 0709 13:04:09.444407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.517428 (+ 73021us) service_pool.cc:224] Handling call
0709 13:04:10.638221 (+1120793us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54001e73d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:10.638416 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1332696320}
I20260709 13:04:10.561915  8651 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:10.676750  8729 raft_consensus.cc:2410] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:04:10.658895  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1200) took 1304 ms. Trace:
I20260709 13:04:10.577153  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:10.677508  8724 rpcz_store.cc:276] 0709 13:04:09.354829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.362183 (+  7354us) service_pool.cc:224] Handling call
0709 13:04:09.362554 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:09.991211 (+628657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.658673 (+667462us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b540022a954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:10.658839 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1433165696}
I20260709 13:04:10.678392  8579 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:10.678817  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 287083us
I20260709 13:04:10.661039  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:10.443725  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:10.680220  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.550714  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:10.336462 14770 raft_consensus.cc:2764] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
I20260709 13:04:10.681000  8652 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:10.511772  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1072) took 1484 ms. Trace:
I20260709 13:04:10.558522  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1074) took 1084 ms. Trace:
I20260709 13:04:10.549273  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.575899  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1196) took 1227 ms. Trace:
I20260709 13:04:10.578944  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3242) took 1470 ms. Trace:
I20260709 13:04:10.591452  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.396122  8725 raft_consensus.cc:2410] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:04:10.658459  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2782) took 1627 ms. Trace:
W20260709 13:04:10.542042  8657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2899) took 1703 ms (client timeout 2175 ms). Trace:
I20260709 13:04:10.485111  9331 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:10.546691  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:10.497835  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.336306 15010 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.686125  8550 leader_election.cc:400] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16.
W20260709 13:04:10.694620  8657 rpcz_store.cc:269] 0709 13:04:08.838852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.838975 (+   123us) service_pool.cc:224] Handling call
0709 13:04:10.541807 (+1702832us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400424854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:10.541978 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1160031104}
I20260709 13:04:10.457070  9326 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:10.655193  8727 rpcz_store.cc:276] 0709 13:04:09.337436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.337683 (+   247us) service_pool.cc:224] Handling call
0709 13:04:10.541699 (+1204016us) spinlock_profiling.cc:243] Waited 75.3 ms on lock 0x7b54001e64d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:10.541870 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":173261568}
I20260709 13:04:10.695884  8655 rpcz_store.cc:276] 0709 13:04:09.108629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.287009 (+178380us) service_pool.cc:224] Handling call
0709 13:04:09.287315 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:09.706350 (+419035us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.578641 (+872291us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54003223d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:10.578870 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853245184}
I20260709 13:04:10.697582  8727 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:10.692719  8658 rpcz_store.cc:276] 0709 13:04:09.473840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.473986 (+   146us) service_pool.cc:224] Handling call
0709 13:04:09.491111 (+ 17125us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:09.668710 (+177599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.558249 (+889539us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b540018fbd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:10.558461 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":356758272}
I20260709 13:04:10.677358  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2903) took 1048 ms. Trace:
I20260709 13:04:10.461023  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:10.700966  8653 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:10.691039  8724 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:10.694504  8726 rpcz_store.cc:276] 0709 13:04:09.348387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.348824 (+   437us) service_pool.cc:224] Handling call
0709 13:04:09.349127 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:09.388896 (+ 39769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.575589 (+1186693us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b54000ea454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:10.575821 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49398144}
I20260709 13:04:10.700587  8651 rpcz_store.cc:276] 0709 13:04:09.629134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.561116 (+931982us) service_pool.cc:224] Handling call
0709 13:04:10.561513 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.676972 (+115459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.677245 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:10.712831  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 322367us
I20260709 13:04:10.719652  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:10.732913  8729 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:10.341475 15008 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:10.694665  8728 rpcz_store.cc:276] 0709 13:04:09.030616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.042861 (+ 12245us) service_pool.cc:224] Handling call
0709 13:04:09.083436 (+ 40575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:09.256584 (+173148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.658229 (+1401645us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b5400338954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:10.658391 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297927936}
W20260709 13:04:10.459223 15046 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:04:10.692517  8659 rpcz_store.cc:276] 0709 13:04:09.027276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.027490 (+   214us) service_pool.cc:224] Handling call
0709 13:04:09.052865 (+ 25375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:09.053284 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:09.053325 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:09.086910 (+ 33585us) log.cc:844] Serialized 26 byte log entry
0709 13:04:09.515682 (+428772us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b18001265e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:09.647488 (+131806us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:09.647544 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.647631 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.511382 (+863751us) raft_consensus.cc:1685] finished
0709 13:04:10.511407 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.511657 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12399,"raft.queue_time_us":39681,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8347,"spinlock_wait_cycles":4334848,"thread_start_us":131631,"threads_started":1,"wal-append.queue_time_us":762209}
I20260709 13:04:10.748176  8651 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:10.498095 14924 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:10.577744 15030 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.002s
I20260709 13:04:10.594710 14893 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:04:10.769625  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50109us
I20260709 13:04:10.843201  8697 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:10.596612 15005 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.533s	user 0.000s	sys 0.006s
I20260709 13:04:10.544271 14765 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.546665 14985 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.472510 14963 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:10.388425 14827 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:10.547717 15053 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:10.824785 14733 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:10.794693  9322 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:10.466423 14962 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.488590 14810 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:10.580077 14505 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:10.563839 15018 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:10.559738 15041 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:10.775931 15010 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.564113 14907 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.561676 14748 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:10.341768 14974 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.544137 14952 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:10.392424 15020 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.369714 14841 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.863572 15053 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.879104 15044 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:10.898059 14841 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:10.898521 15044 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.899318  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:10.899766  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2586) took 1500 ms. Trace:
I20260709 13:04:10.900597  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:10.900125  8576 rpcz_store.cc:276] 0709 13:04:09.399303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.416328 (+ 17025us) service_pool.cc:224] Handling call
0709 13:04:09.416600 (+   272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:09.712687 (+296087us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:09.712720 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:09.718888 (+  6168us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.205704 (+486816us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:10.205786 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.205882 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.899566 (+693684us) raft_consensus.cc:1685] finished
0709 13:04:10.899587 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.899714 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33273,"raft.queue_time_us":76217,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":5453,"thread_start_us":672351,"threads_started":2,"wal-append.queue_time_us":1079335}
I20260709 13:04:10.849581 14765 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.905381  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:10.908682  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:10.909368  8582 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:10.910513  8652 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:10.911825  8657 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:10.912561  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:10.874413 15018 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:10.776808 15017 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.841553  8730 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
W20260709 13:04:10.821203 15021 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.729s	user 0.006s	sys 0.003s
I20260709 13:04:10.748817 14926 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.748111 15008 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.685461 14989 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:10.846964  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 487873us
W20260709 13:04:10.922819  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.399874 15003 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.419989 14821 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:10.924088 14267 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.067s	sys 0.127s
W20260709 13:04:10.924326 14991 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.447s	user 0.013s	sys 0.014s
I20260709 13:04:10.749002 14769 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:10.924893 14991 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.451s	user 0.013s	sys 0.014s
I20260709 13:04:10.833298 15030 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:10.874141  8579 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:10.747426  8732 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:04:10.925772  8732 pending_rounds.cc:85] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 7
I20260709 13:04:10.926047  8732 pending_rounds.cc:107] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:04:10.857240 14942 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.857465 14963 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:10.845263 14893 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:10.885388  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:10.874106 14962 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.873946 14505 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:10.918350 14970 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:10.888196 14748 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.899250 15053 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:10.910837 15020 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:10.466893 14947 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.946576 14959 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:10.587502 15019 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:10.914229  9326 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:10.916174  9328 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:10.925932 15008 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:10.882771 14827 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:04:10.881902 14770 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:10.920311  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.845253 15005 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:10.924476 14267 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.067s	sys 0.127s
I20260709 13:04:10.864010 14985 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.842299 14924 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.926721  8732 raft_consensus.cc:2889] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:10.923136  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:10.930378  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.934537  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:10.901382 15044 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.935456  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2584) took 1538 ms. Trace:
I20260709 13:04:10.929010  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:10.940667  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:11.056820  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 127894us
I20260709 13:04:10.877434 14810 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.941846  9323 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:10.925166 14991 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:10.943311  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.073822  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1071) took 2050 ms. Trace:
I20260709 13:04:11.074321  9327 raft_consensus.cc:2410] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:04:10.895205 14974 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:10.944414  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59160.8us
I20260709 13:04:11.074965 14961 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:11.074375 14991 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:11.076241 14899 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.851s	user 0.002s	sys 0.009s
I20260709 13:04:11.077474  9294 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:10.907603 14765 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.940513 14942 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.948761 14963 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.949998 14893 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.950795 14962 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.955121 14748 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:10.881059 15041 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.878952 14733 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.907618 14841 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:04:10.858855  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:10.963569  8625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.012s	sys 0.014s
W20260709 13:04:11.116035  8625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.013s	sys 0.014s
W20260709 13:04:11.116394  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 742773us
I20260709 13:04:11.117921  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:10.942786 14821 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:10.976459  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.985791  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2906) took 1325 ms. Trace:
I20260709 13:04:11.120371  8654 rpcz_store.cc:276] 0709 13:04:09.660629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.221987 (+561358us) service_pool.cc:224] Handling call
0709 13:04:10.978290 (+756303us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b5400059bd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:10.985702 (+  7412us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":375126784}
I20260709 13:04:10.986706  8658 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:04:10.956233 14970 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.930832 15021 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:10.926556 15017 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:11.004972 14906 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.009s	sys 0.020s
I20260709 13:04:11.005172 15040 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.016741 15003 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.017949 14827 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.146492  8580 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:09.590113  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:11.018517 14884 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:11.019857 15055 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 13:04:11.021102  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3163) took 1083 ms. Trace:
I20260709 13:04:11.020148  8474 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 12, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 12 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:11.125656 14970 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.022929 15005 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:10.969180 15018 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.005970 14770 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:11.031116 15048 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 13:04:11.036000 14505 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.969632 14947 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.039721  9326 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:10.935556 14926 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:10.931035 14989 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.056901  8579 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:10.940785  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.062309 14924 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.066423 14980 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:11.066607  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:11.067806 14688 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:11.068500 15064 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:11.071404 14985 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.071516 14938 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:11.072705  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2905) took 1422 ms. Trace:
I20260709 13:04:11.074365  8656 rpcz_store.cc:276] 0709 13:04:09.023190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.033540 (+ 10350us) service_pool.cc:224] Handling call
0709 13:04:10.541940 (+1508400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.659245 (+117305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.073545 (+414300us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400424850. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:11.073739 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3373009152}
W20260709 13:04:11.079397  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 441211us
I20260709 13:04:11.082161  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.084213  9331 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:11.055171  8584 rpcz_store.cc:276] 0709 13:04:09.397329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.440186 (+ 42857us) service_pool.cc:224] Handling call
0709 13:04:09.440442 (+   256us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:09.449292 (+  8850us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:09.449321 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:09.467104 (+ 17783us) log.cc:844] Serialized 52 byte log entry
0709 13:04:09.713164 (+246060us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:09.713223 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.713300 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.912736 (+1199436us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b240011e7d8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b67fb0 00007f14097de488 00007f14097debe8 00007f14097b6398 00007f14097b176a 00007f14097b0bb6
0709 13:04:10.927449 (+ 14713us) raft_consensus.cc:1685] finished
0709 13:04:10.927478 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.935391 (+  7913us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10204,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":6072,"spinlock_wait_cycles":22968064,"thread_start_us":222758,"threads_started":1,"wal-append.queue_time_us":400194}
I20260709 13:04:11.165215 15056 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:11.085546  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 242403us
I20260709 13:04:11.182061 15056 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.182920  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:11.112852  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.184286  8697 leader_election.cc:400] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15.
I20260709 13:04:11.184650  8697 leader_election.cc:403] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 13:04:11.184738  9328 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:11.075471 14961 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:11.185968 14985 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15.
I20260709 13:04:11.186079 15041 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.126860  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2902) took 1498 ms. Trace:
I20260709 13:04:11.186836  8650 rpcz_store.cc:276] 0709 13:04:09.628883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.551350 (+922467us) service_pool.cc:224] Handling call
0709 13:04:10.551782 (+   432us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:10.552243 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.552283 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:10.552996 (+   713us) log.cc:844] Serialized 52 byte log entry
0709 13:04:10.874327 (+321331us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:11.124310 (+249983us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.126525 (+  2215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.126605 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.126648 (+    43us) raft_consensus.cc:1685] finished
0709 13:04:11.126660 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.126797 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":816,"raft.queue_time_us":11877,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":6185,"thread_start_us":555800,"threads_started":2,"wal-append.queue_time_us":510453}
W20260709 13:04:11.129931  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:11.130702  8733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.006s	sys 0.016s
W20260709 13:04:11.188776  8733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.006s	sys 0.016s
I20260709 13:04:11.189985 15021 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.134289 14967 consensus_queue.cc:1048] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:11.191202 14770 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.134784  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:11.192371  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57677.3us
I20260709 13:04:11.192418  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:11.192797 15005 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:11.193558 15005 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:11.194844 15005 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.193354 14985 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:11.135188 14960 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:11.146756 15040 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.199608  8729 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:11.199934 15040 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.127979  8576 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:11.114487 14899 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.200793  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.021324 15010 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.126765 14898 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:11.201639  8726 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:04:11.202052 14898 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:11.202411  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.165639 15061 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.203193 14898 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.203847  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.125633 14748 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:11.179692 14938 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:11.208699  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:11.209753 14938 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.210402  8651 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:11.211068 14938 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.138410 14810 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.128175 15044 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:04:11.213568 14893 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:11.138835 14906 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:11.185478 15067 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:11.123167  9330 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:11.123618 14733 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.172636  9324 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:11.173213  8577 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:11.201172 14899 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.173870 14947 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.174961  9329 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:11.178193 14980 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:11.178970 15059 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:11.208228 15018 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:11.259383  8659 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:11.225667 15067 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.260632  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.260939 15067 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.179318 15064 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:11.147925  8578 rpcz_store.cc:276] 0709 13:04:09.937725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.020784 (+ 83059us) service_pool.cc:224] Handling call
0709 13:04:10.043065 (+ 22281us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.043427 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:10.397774 (+354347us) log.cc:844] Serialized 14 byte log entry
0709 13:04:10.872900 (+475126us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.880132 (+  7232us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.880312 (+   180us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:10.880363 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.880550 (+   187us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.020874 (+140324us) raft_consensus.cc:1685] finished
0709 13:04:11.020901 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.021033 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4809,"prepare.queue_time_us":94712,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":4277,"raft.queue_time_us":2800,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":15110,"thread_start_us":738849,"threads_started":2,"wal-append.queue_time_us":566104}
I20260709 13:04:11.207952  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2603) took 1162 ms. Trace:
I20260709 13:04:11.184537 15056 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.285936  8581 rpcz_store.cc:276] 0709 13:04:10.045830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.237718 (+191888us) service_pool.cc:224] Handling call
0709 13:04:10.238060 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.253055 (+ 14995us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.253086 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.253581 (+   495us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.685040 (+431459us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:10.685084 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.685173 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.903610 (+218437us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b48002dffd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:04:11.207630 (+304020us) raft_consensus.cc:1685] finished
0709 13:04:11.207652 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.207863 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14455,"raft.queue_time_us":15129,"raft.run_cpu_time_us":3385,"raft.run_wall_time_us":589793,"spinlock_wait_cycles":25867136,"thread_start_us":431151,"threads_started":1,"wal-append.queue_time_us":610843}
I20260709 13:04:11.179826  8657 rpcz_store.cc:276] 0709 13:04:09.650160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.755438 (+1105278us) service_pool.cc:224] Handling call
0709 13:04:10.911667 (+156229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.912185 (+   518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.072452 (+160267us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b540026c350. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:11.072626 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":358357248}
I20260709 13:04:11.225517 14442 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:11.217559  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 151040us
I20260709 13:04:11.213225  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.180241 14962 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:11.201333  8732 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:11.307664  8732 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:11.308792  8732 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.310039  8732 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:11.222389  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.152348 14950 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:11.226186  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:11.229374  8654 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:11.250619  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2602) took 1260 ms. Trace:
I20260709 13:04:11.257009  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.235365  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:11.316596  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 90492.6us
I20260709 13:04:11.316826  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:11.316186  8583 rpcz_store.cc:276] 0709 13:04:09.989806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.220467 (+230661us) service_pool.cc:224] Handling call
0709 13:04:10.220822 (+   355us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:10.221271 (+   449us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.221323 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:10.228283 (+  6960us) log.cc:844] Serialized 52 byte log entry
0709 13:04:10.526948 (+298665us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:10.527012 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.527112 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.250280 (+723168us) raft_consensus.cc:1685] finished
0709 13:04:11.250305 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.250550 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8280,"raft.queue_time_us":1169,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":2407,"thread_start_us":133247,"threads_started":1,"wal-append.queue_time_us":434340}
I20260709 13:04:11.226619  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:11.321684  8724 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.293319  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:11.322430  8657 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:11.261274  8659 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:11.324915  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:11.237210  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:11.174613  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:11.329816  8579 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:11.290066  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.331015  8656 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:11.291291  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.167577 14942 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:11.206954 15017 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.179101 14688 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:11.223122  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:11.244747 15060 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.152629 14827 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.306732  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:11.311939  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.373030  8651 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:11.171777 14505 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.332425  8726 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:11.380604  8654 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:11.313510  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:11.381824  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:11.382306  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 68935.7us
I20260709 13:04:11.382855  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:11.384476  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:11.247828 14899 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.257791 14980 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:11.262992 14267 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.342406  8658 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:11.336263  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:11.287488 15064 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:11.359223  8652 raft_consensus.cc:2410] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:04:11.346329 15040 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:11.358400  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2807) took 1111 ms. Trace:
I20260709 13:04:11.423548  8731 rpcz_store.cc:276] 0709 13:04:10.247154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.725860 (+478706us) service_pool.cc:224] Handling call
0709 13:04:10.819115 (+ 93255us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.819530 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.819571 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.820371 (+   800us) log.cc:844] Serialized 26 byte log entry
0709 13:04:11.008670 (+188299us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:11.009179 (+   509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.009702 (+   523us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.358148 (+348446us) raft_consensus.cc:1685] finished
0709 13:04:11.358172 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.358333 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123878,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":19540,"thread_start_us":186343,"threads_started":1,"wal-append.queue_time_us":359886}
W20260709 13:04:11.368997  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 407192us
I20260709 13:04:11.270398 14967 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.078s
I20260709 13:04:11.228363 14841 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.096s
W20260709 13:04:11.376502  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 155179us
I20260709 13:04:11.323112 14442 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.408366  8732 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
W20260709 13:04:11.400940  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 64772us
I20260709 13:04:11.390678 14976 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:11.411208  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:11.363226  8724 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
W20260709 13:04:11.393080 14948 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.009s	sys 0.020s
I20260709 13:04:11.344964 14950 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:11.430438  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.455559  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:11.435168  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:11.248615 15008 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.438089  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.203902 15061 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.341603 14959 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:11.444092  8732 pending_rounds.cc:85] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 12
I20260709 13:04:11.440656  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:11.321590 15059 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.106s
I20260709 13:04:11.400875 15019 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:04:11.430912 15042 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 13:04:11.454347  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:11.422360 15009 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:11.453727 15056 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:04:11.427206 15021 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:11.457309  8658 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:11.424149 15068 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:11.469300  8732 pending_rounds.cc:107] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:04:11.381605 15060 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.468209  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:11.473037  8575 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.038s	user 0.010s	sys 0.018s
I20260709 13:04:11.363273 14906 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.464921 14765 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.448349 15040 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:11.504328  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:11.422991 14267 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.480237  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.427954 14977 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
W20260709 13:04:11.504674  8575 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.085s	user 0.010s	sys 0.018s
I20260709 13:04:11.474197 15009 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.472608 14924 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.480594  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.526535  8659 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:11.457518 14899 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:11.453758 15073 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.505379 15068 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.533627 15068 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:11.534119 14267 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.534710 14267 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:11.534814 15008 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.535189 14267 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.442822 14962 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:04:11.537839 14948 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:11.538292 14948 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:11.538448 14962 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.538777 14948 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:11.539002 14962 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.450907 15017 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 13:04:11.542596 14810 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:04:11.520455  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52332.6us
I20260709 13:04:11.480660 14976 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:04:11.549342 14899 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:11.551486  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.551406 14899 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.541333 14962 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.556586  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.546869 14924 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.513706 15021 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:11.556231 15040 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.464699 15041 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:11.557724  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.557854 15041 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.559581 14733 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:11.513876 14985 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:11.551734  9323 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:11.561136 14827 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.482532 15069 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 13:04:11.558811 14924 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.562588 15069 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:11.563058 15069 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:11.563694  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:11.550307 14989 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.465950 14505 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:11.565304 14505 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.565369  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:11.565939 14505 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.566293  8658 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:11.531812 15078 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.531893 14926 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.532655  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.601908  8659 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:11.533306 15073 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.525883 15056 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:11.467496 14980 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.564020 15069 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.482959 14841 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:11.616210  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:11.567782  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.548374  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 2981) took 1307 ms. Trace:
I20260709 13:04:11.617301  8733 rpcz_store.cc:276] 0709 13:04:10.241185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.598278 (+357093us) service_pool.cc:224] Handling call
0709 13:04:10.598659 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:10.599062 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.599101 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.599685 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 13:04:11.189148 (+589463us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:11.189206 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.189287 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.548116 (+358829us) raft_consensus.cc:1685] finished
0709 13:04:11.548137 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.548308 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":107474,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":69404,"thread_start_us":589079,"threads_started":1,"wal-append.queue_time_us":838736}
I20260709 13:04:11.619751  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.620718  8733 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:11.622314  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.625504  8733 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 18 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:04:11.516913 14765 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.575668  8726 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:11.572527  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:11.630689  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 58248.9us
I20260709 13:04:11.630827  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.581118  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.583328  8657 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:11.505056  8732 raft_consensus.cc:2889] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:11.565153 15064 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.602694  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.454931 14688 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:11.605968  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 374472us
I20260709 13:04:11.525544 15009 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.507691 14952 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.068s
I20260709 13:04:11.610811  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.568786 15068 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.516284 15010 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.089s
I20260709 13:04:11.623562 14267 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.628455  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:11.710608  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:11.524639  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 447211us
I20260709 13:04:11.580734  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.632555 14442 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.632855 14926 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:11.615861 14970 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.633692 14841 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:11.560328 14950 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.669488  9325 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:11.728906  9331 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:11.670220  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.729058  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.729380  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.729773  9330 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:11.730512  8651 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:11.730786  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.732172  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.672144 15060 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:11.616173  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.680753  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.743410  8732 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:11.744652  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.560153 15041 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.705803  8657 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:11.712502  8733 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:11.747784  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.748602  8733 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:11.713927  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.716234  9328 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:11.635365 15071 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.751187 15071 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.721230  8697 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:04:11.723413  8655 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.003s	sys 0.034s
W20260709 13:04:11.752948  8655 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.003s	sys 0.035s
I20260709 13:04:11.754091 14267 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:11.754549 14267 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.754906 14267 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:11.756575  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.672693 14980 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.759267  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.636557  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:11.759932  8732 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
W20260709 13:04:11.760445  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 123976us
I20260709 13:04:11.760875  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.761605  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:11.672482 15069 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:11.726316  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:11.716902 14926 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.765702 14926 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:11.767748  8732 raft_consensus.cc:2410] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:04:11.768769  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.725220 14442 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.769578  8732 raft_consensus.cc:2410] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 14 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 13:04:11.769665 14442 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.770443  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.771800 14442 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:11.772965 15056 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:11.678515  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:11.774578  8726 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:11.775758  8726 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:11.776561  8733 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:11.608731 15078 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.737443  9325 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:11.777711 14960 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:11.778257 14960 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:11.778653 15078 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.779166 14960 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.785454 14924 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.786149 14924 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.788746 14924 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.790848 14970 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.795842 15068 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.797920 14841 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.800182 15073 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.704818 15018 consensus_queue.cc:1059] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:04:11.810190  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.810364  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.810969  9325 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:11.811568  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.811892  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.812366 14765 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:11.814162  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.815088 15083 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:11.815516 15083 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:11.815691 14893 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.000s
I20260709 13:04:11.725188 14688 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:11.824137  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1075) took 1568 ms. Trace:
I20260709 13:04:11.828603  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.829609  9328 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:11.825977  8655 rpcz_store.cc:276] 0709 13:04:10.255945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.746968 (+491023us) service_pool.cc:224] Handling call
0709 13:04:10.747299 (+   331us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:10.817153 (+ 69854us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.817191 (+    38us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:04:10.818119 (+   928us) log.cc:844] Serialized 78 byte log entry
0709 13:04:11.140181 (+322062us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:11.753438 (+613257us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.823529 (+ 70091us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.823756 (+   227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.823851 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.823899 (+    48us) raft_consensus.cc:1685] finished
0709 13:04:11.823915 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.824068 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69651,"prepare.queue_time_us":462012,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":43764,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":11041,"thread_start_us":871796,"threads_started":2,"wal-append.queue_time_us":629286}
I20260709 13:04:11.838735  8654 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:11.848152  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:11.849073  9323 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
W20260709 13:04:11.851127  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 384829us
I20260709 13:04:11.821696 15061 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:11.817152 14267 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.856910  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:11.816347 15083 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.846199 14991 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.887758  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:11.871654  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.888695  9323 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:11.846338 14924 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.845985 15041 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:11.906725  8733 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:11.908730  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:11.909965 14980 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:11.910677 14980 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:11.911800 14980 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.857473 14959 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:11.832677 14926 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.881479 14898 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:11.907339 14821 raft_consensus.cc:3055] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:11.866303 15018 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 13:04:11.921078 15005 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.952355  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2605) took 1769 ms. Trace:
I20260709 13:04:11.936425 15083 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:11.964854  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.945709 14898 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:11.965449 14898 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:11.954084 14765 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:11.965764 14505 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.966277 14765 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.969590 14505 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.966958 15064 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:11.973026  8623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.012s	sys 0.009s
W20260709 13:04:11.973662  8623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.013s	sys 0.009s
W20260709 13:04:11.974545  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 520235us
I20260709 13:04:11.917760 14924 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:11.949362 14821 raft_consensus.cc:740] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:12.026729  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:12.028299  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:11.920526 14991 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.031940 14991 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:11.930194 15041 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:11.976171 15056 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:12.001556  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 267913us
I20260709 13:04:12.033773  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:12.038380  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:11.991500  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 487149us
I20260709 13:04:12.042186  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:11.972460 15005 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.050143  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.052109  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.053040  8577 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
W20260709 13:04:11.918849 14952 consensus_queue.cc:1175] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a0d703ee18464475a290425976246555" responder_term: 19 status { last_received { term: 17 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:04:12.054426  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:11.994067 15053 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.029s	sys 0.006s
I20260709 13:04:12.055308  8579 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:04:12.055431 15053 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.029s	sys 0.006s
W20260709 13:04:12.056849 15094 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 13:04:12.057761 15102 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:04:12.058174 15088 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 13:04:12.058701 15061 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:12.059470 15009 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.059556 14899 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:12.060811  8584 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:12.059963 15104 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.074060  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.077296  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:12.077991  8729 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.054426  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.055229  8581 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:04:12.081243  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:12.082856  8581 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.077924  8577 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:12.073087  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.085630  8580 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:04:12.085665  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:12.086894  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.092455  8582 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:12.087133  8576 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:12.094676  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:12.085758  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:12.091143  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:12.091606  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.098856  8579 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.099442  8578 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:04:12.073424  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:12.102828  8583 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.061295 15061 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:12.100565  8577 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.061594 14962 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:12.061928 15009 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.080255 15017 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:12.114326  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 404485us
I20260709 13:04:12.110474  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:12.120944  9323 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.120968  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.122524  8650 raft_consensus.cc:2410] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 18 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:04:12.118964  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:12.123127  9322 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.086834 15040 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:11.991648 14959 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.091326 15050 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:12.137445  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.062347 14926 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.135095  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:12.080786 15056 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.149971  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:12.151633  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:12.189962  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:12.190549  8656 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.152220  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:12.189157  8655 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:12.091144 15095 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 13:04:12.161003 14906 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:04:12.162283 15089 log.cc:927] Time spent T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 13:04:12.169428  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.170224  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:11.958982  8575 rpcz_store.cc:276] 0709 13:04:10.183204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.358454 (+175250us) service_pool.cc:224] Handling call
0709 13:04:10.358714 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.359861 (+  1147us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:11.527556 (+1167695us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.951851 (+424295us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:11.951939 (+    88us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:11.951998 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.952082 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.952261 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":177418,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":111,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":5539,"thread_start_us":1521684,"threads_started":2}
I20260709 13:04:12.304277  8582 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:12.132442  8583 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:12.338488  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.339130  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.340113  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.340358  8583 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:12.309058  8579 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:12.167876  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:12.342569  8650 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:12.342438  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.343837  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.344939  8650 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:12.168675  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.191561  8623 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:12.346251  8652 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
W20260709 13:04:12.347750  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 156178us
I20260709 13:04:12.348183  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.343830  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.146699  8581 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:12.349740  8579 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:12.307884  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.351158  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.351694  8623 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:12.352447  8650 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:04:12.193022  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58040.3us
I20260709 13:04:12.353976  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.330439  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.334719  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3264) took 1118 ms. Trace:
I20260709 13:04:12.336366  8474 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 11, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 11 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:12.355108  8654 rpcz_store.cc:276] 0709 13:04:11.215685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.381540 (+165855us) service_pool.cc:224] Handling call
0709 13:04:12.334175 (+952635us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:12.334428 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":969032960}
I20260709 13:04:12.182572  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.335687  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.358245  8659 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:12.358799  8657 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:04:12.359766  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.360052  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.360793  8659 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:12.347908  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.337672  8658 raft_consensus.cc:2393] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:12.361860  8652 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:12.357261  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.307947  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:12.365592  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2814) took 1166 ms. Trace:
I20260709 13:04:12.365717  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.366230  8728 rpcz_store.cc:276] 0709 13:04:11.199173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.200593 (+  1420us) service_pool.cc:224] Handling call
0709 13:04:11.200929 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:11.201294 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:11.201320 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:11.201745 (+   425us) log.cc:844] Serialized 26 byte log entry
0709 13:04:11.608757 (+407012us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:11.608800 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.608869 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:12.364953 (+756084us) raft_consensus.cc:1685] finished
0709 13:04:12.365000 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.365288 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1873,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":71,"spinlock_wait_cycles":25344,"thread_start_us":406679,"threads_started":1,"wal-append.queue_time_us":685765}
I20260709 13:04:12.368227  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.176580  8729 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:12.370146  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.371479  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:12.371932  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.372491  8731 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.372916  8729 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:12.375422  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:12.377072  8622 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:12.369303  8726 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:12.380008  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:12.383040  8726 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.383937  8731 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:12.380136  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2827) took 1143 ms. Trace:
I20260709 13:04:12.384749  8724 rpcz_store.cc:276] 0709 13:04:11.236285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.455093 (+218808us) service_pool.cc:224] Handling call
0709 13:04:11.455514 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:11.627008 (+171494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:11.627041 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:11.634346 (+  7305us) log.cc:844] Serialized 26 byte log entry
0709 13:04:11.953776 (+319430us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:11.953839 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.379490 (+425651us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54001eafd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:12.379557 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:12.379621 (+    64us) raft_consensus.cc:1685] finished
0709 13:04:12.379634 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.379961 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1167,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":39142,"spinlock_wait_cycles":393411456,"thread_start_us":318993,"threads_started":1,"wal-append.queue_time_us":421558}
I20260709 13:04:12.341114  8575 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:12.381227  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.389051  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:12.391549  8729 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:12.377110  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.188504  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:12.191633  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:04:12.486377  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 294630us
I20260709 13:04:12.212248  9322 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:12.214782  9323 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:12.496614  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1083) took 1274 ms. Trace:
I20260709 13:04:12.159850 14267 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.228798 14765 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.148197 14821 consensus_queue.cc:260] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.370061  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:12.499543  8584 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:12.187654 14893 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.502259 14893 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.503186 14770 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:12.503789 14770 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:12.168916 14898 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.179203 14924 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:12.233048 14948 consensus_meta.cc:287] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.694s	user 0.012s	sys 0.000s
I20260709 13:04:12.008361 14961 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.253253 14963 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:12.197201 14970 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 13:04:12.186297 15005 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.154708 15040 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.157292 15050 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.184695 15041 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.240612 15056 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.155124 15061 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
W20260709 13:04:12.241659 15067 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.980s	user 0.007s	sys 0.011s
I20260709 13:04:12.244020 15071 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.197479 15102 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:12.115868 15103 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:12.088694 15104 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.180717 15122 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.244696 15058 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:12.412127  8656 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:12.393953  8474 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 11 to 14, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 14 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:12.428680  8655 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
W20260709 13:04:12.432230  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55575.3us
I20260709 13:04:12.434135 14926 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:12.441777  8577 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:12.443473 15021 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.458288 14947 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:12.458583  8726 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:04:12.461894 15019 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.462332 15020 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:12.463178 15100 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:12.463364 15130 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:12.462353  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:12.464957 15137 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:04:12.465533  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.466089  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:12.468892 14733 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:12.491570  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:12.505786  8650 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
W20260709 13:04:12.693593  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 204485us
W20260709 13:04:12.694207  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 331640us
I20260709 13:04:12.535562  8730 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:12.444890 15010 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:12.532874  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:12.690389 15020 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.693054 14893 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.536741 14947 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:12.518301 15040 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.508184 14898 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.526121 15103 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:12.525308 15102 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.528636 15104 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.529971 15058 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:12.695976 15137 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.520632 15050 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.536542 15021 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:12.718704  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 256467us
I20260709 13:04:12.529585 15122 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.715586 15020 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:12.497311  8651 rpcz_store.cc:276] 0709 13:04:11.222459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.731376 (+508917us) service_pool.cc:224] Handling call
0709 13:04:11.731719 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:11.731960 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:11.822649 (+ 90689us) log.cc:844] Serialized 14 byte log entry
0709 13:04:12.416406 (+593757us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:12.434999 (+ 18593us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:12.495877 (+ 60878us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.496060 (+   183us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.496348 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2856,"prepare.queue_time_us":91237,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":3096,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":48,"thread_start_us":576879,"threads_started":2,"wal-append.queue_time_us":421732}
I20260709 13:04:12.533484 14926 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.509255 14948 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.523861 15067 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.723861 15010 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:12.537761 15019 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.748653 14980 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:12.748901 14991 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:12.714891 14733 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.531514  8474 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 5 to 7, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 7 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:12.751297  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 285631us
I20260709 13:04:12.762719 14961 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:12.773166  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 76382.8us
W20260709 13:04:12.764158  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 588342us
I20260709 13:04:12.764780 15018 raft_consensus.cc:2764] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:04:12.765688 15104 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:12.734052 15102 raft_consensus.cc:3060] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:12.505364 14770 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.730677 15137 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.777053  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.749219 14980 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:12.792474  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 301181us
I20260709 13:04:12.750617 15068 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:12.756330 14926 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.752825 15122 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.788902  8584 raft_consensus.cc:2468] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:12.794314  8729 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:12.750392 15054 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.794399 14980 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:12.796329 14959 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.796446 15061 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:12.811004  9328 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
W20260709 13:04:12.797448  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 74223.9us
I20260709 13:04:12.811204 14959 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.812999 14924 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:12.813827 14924 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.811182 14267 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:12.815428  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 626808us
I20260709 13:04:12.815277 14267 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.816344 14924 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.817090  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:12.817566 14821 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.749341 14991 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:12.817647 14748 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:12.818511 14991 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.749650 15071 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:12.822037 14906 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:12.814044 15058 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.822940 15019 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.823221 15058 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:12.823469  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:12.824436 14770 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.824896 15041 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:12.821704 15071 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.820871  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:12.797739 15103 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.794674 15068 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:12.833973 15061 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:12.834512 15068 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:12.835510 14688 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 1.009s	user 0.013s	sys 0.005s
I20260709 13:04:12.841764  9324 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:12.843408  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:12.843883  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1367) took 1291 ms. Trace:
I20260709 13:04:12.844317  9324 rpcz_store.cc:276] 0709 13:04:11.552714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.556289 (+  3575us) service_pool.cc:224] Handling call
0709 13:04:12.843644 (+1287355us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54003b01d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:12.843782 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2943078272}
W20260709 13:04:12.845737  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.846318  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:12.822495 14906 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.847620 14906 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:12.849725 15017 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:12.850606 15017 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.854059 14827 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:12.855109 14827 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:12.855733  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:12.797441 15067 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.845136  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:12.750257 15010 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.786818 15083 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.106s
I20260709 13:04:12.782300 14947 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.804392 14926 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:12.855778 14827 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:12.866007  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.866366 14810 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:04:12.856930 15054 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.859869 15101 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:12.868161 14733 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:12.867765 14991 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:12.872025  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 339290us
I20260709 13:04:12.853188 15017 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.878088  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.880632  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:12.882761  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.883090 14827 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.884136 15054 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:12.885944  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.830742 15008 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.893605  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.895588  8698 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:12.897759  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.898028 14267 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.898198 15021 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:12.899459  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:12.902305  8654 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:04:12.903211  8650 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:12.905494 14733 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:12.905633 14959 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.906226 14733 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.907507  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:12.908644 14733 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.909799 15101 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:12.910347 15101 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:12.910794 15101 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:12.911996  8474 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 14. New cstate: current_term: 14 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:12.914523  8474 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 13, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 13 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:12.924410  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:12.887625 14947 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:12.926558  8476 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 307) took 45172 ms (client timeout 15000 ms). Trace:
I20260709 13:04:12.926683  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.929283 15008 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.932083 15008 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.933900 14926 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.935163 15097 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.946856 15097 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:12.935729  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:12.936527  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65316us
W20260709 13:04:12.949944  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 121802us
I20260709 13:04:12.951318  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:12.952420  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59639.6us
W20260709 13:04:12.953591  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58111.6us
I20260709 13:04:12.955317  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:12.955797 14962 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.956610 15060 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:12.957139 15060 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:12.957697 15060 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:12.956434 14962 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.961930  8725 raft_consensus.cc:2410] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:04:12.963363 14898 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:12.963894 14898 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:12.964319 14898 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:12.966228 15021 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:12.941071  8730 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:04:12.967885 15137 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:12.968130  9294 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:12.956382 15130 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:12.970547 14948 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:12.897203 15071 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:12.972975 14962 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:12.964286  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:04:12.976089 15141 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:04:12.977550 15097 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.978798 14991 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.979918 14267 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:12.981575 15058 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:12.982302 15058 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:12.983384  8726 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:04:12.983629 15058 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:12.985065  8697 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:12.945012 14926 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:12.986697  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:12.982304 14267 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:12.990008  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 52028.2us
I20260709 13:04:12.995867 15040 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:12.997129 15021 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:12.997674 15021 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:12.997751  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46564.4us
I20260709 13:04:12.998862  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:12.999589 14810 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:12.999894 15021 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.001452 14810 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:12.963913  8725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2815) took 1764 ms (client timeout 1710 ms). Trace:
W20260709 13:04:13.004994  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49532.2us
W20260709 13:04:13.003684  8725 rpcz_store.cc:269] 0709 13:04:11.199484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.202184 (+  2700us) service_pool.cc:224] Handling call
0709 13:04:12.963581 (+1761397us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b5400436054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:12.963788 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4043483520}
W20260709 13:04:13.015357  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48185.7us
W20260709 13:04:13.005772 15142 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.123s	user 0.002s	sys 0.000s
I20260709 13:04:12.990020 14267 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:13.016784  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:13.009591 15021 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:13.059362  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 104220us
I20260709 13:04:13.054629  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:13.059748  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.054627  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:13.066517  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49836.9us
W20260709 13:04:13.085705  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 87404.1us
W20260709 13:04:13.111908  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 58634.9us
I20260709 13:04:13.047950 14906 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.108517  9322 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
W20260709 13:04:13.122115  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 50041.8us
I20260709 13:04:13.123481  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:13.124362  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 105989us
I20260709 13:04:13.124739  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:13.125049  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:13.129091  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.129520  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.103937 15040 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:13.130511  8474 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 15, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 15 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:13.132028  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 147226us
W20260709 13:04:12.927263  8476 rpcz_store.cc:269] 0709 13:03:27.753921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.754444 (+   523us) service_pool.cc:224] Handling call
0709 13:04:12.920455 (+45166011us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:03.975971 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:03.976686 (+   715us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:03.976744 (+    58us) write_op.cc:435] Acquired schema lock
0709 13:04:03.976782 (+    38us) tablet.cc:662] Decoding operations
0709 13:04:04.013648 (+ 36866us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:04.013749 (+   101us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:04.013777 (+    28us) tablet.cc:685] Acquiring locks for 15 operations
0709 13:04:04.015053 (+  1276us) tablet.cc:701] Row locks acquired
0709 13:04:04.015112 (+    59us) write_op.cc:260] PREPARE: finished
0709 13:04:04.015297 (+   185us) write_op.cc:270] Start()
0709 13:04:04.015464 (+   167us) write_op.cc:276] Timestamp: P: 1783602244015222 usec, L: 0
0709 13:04:04.015504 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 13:04:04.016490 (+   986us) log.cc:844] Serialized 5876 byte log entry
0709 13:04:04.951895 (+935405us) op_driver.cc:464] REPLICATION: finished
0709 13:04:05.249225 (+297330us) write_op.cc:301] APPLY: starting
0709 13:04:05.284646 (+ 35421us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:05.576808 (+292162us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:05.614103 (+ 37295us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:05.842105 (+228002us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:05.852955 (+ 10850us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 13:04:05.853028 (+    73us) write_op.cc:312] APPLY: finished
0709 13:04:05.858527 (+  5499us) log.cc:844] Serialized 139 byte log entry
0709 13:04:06.079229 (+220702us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:06.149700 (+ 70471us) write_op.cc:454] Released schema lock
0709 13:04:06.173612 (+ 23912us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:06.174237 (+   625us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:06.174272 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3454002,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":259093,"apply.run_cpu_time_us":137222,"apply.run_wall_time_us":947113,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66797,"num_ops":15,"prepare.queue_time_us":558637,"prepare.run_cpu_time_us":36160,"prepare.run_wall_time_us":185824,"raft.queue_time_us":157936,"raft.run_cpu_time_us":11146,"raft.run_wall_time_us":228552,"replication_time_us":904242,"thread_start_us":684757,"threads_started":5,"wal-append.queue_time_us":646734,"wal-append.run_cpu_time_us":16547,"wal-append.run_wall_time_us":2752384}]]}
I20260709 13:04:13.138826 15102 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.141278 15068 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.086637 15019 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:13.164848  8653 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
W20260709 13:04:13.175902  8698 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.713s (SENT) [suppressed 8 similar messages]
I20260709 13:04:13.181146  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:13.186524  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:13.188529  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.198777  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:13.081027 14688 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:13.121773 14962 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:13.218881 14980 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.220185  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:13.086274 14898 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.113148 14733 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:13.087056 15097 raft_consensus.cc:2764] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:04:13.227830  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:13.231511  9324 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:13.235492  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:13.236955  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 129018us
I20260709 13:04:13.245581  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:13.224567 15041 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:13.292541  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.256137 15019 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.293994  9328 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:13.295945  9327 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:13.296252  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.297960  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:13.304280  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3263) took 2065 ms (client timeout 1713 ms). Trace:
W20260709 13:04:13.306202  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47465.3us
I20260709 13:04:13.292611  9329 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:13.264292 14770 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:13.304841  8653 rpcz_store.cc:269] 0709 13:04:11.215342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.306424 (+ 91082us) service_pool.cc:224] Handling call
0709 13:04:13.280024 (+1973600us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b540025ba54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:13.280185 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4099008256}
I20260709 13:04:13.090287  8724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:13.310647  8724 raft_consensus.cc:606] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.311121  8724 raft_consensus.cc:614] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:13.312734 15044 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.313378 15044 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.317214  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.318836  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.319543 14906 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:13.320135  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.320974  9324 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:13.323975 15137 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.325544 15040 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:13.326143  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1407) took 1174 ms. Trace:
I20260709 13:04:13.312940  9322 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 20: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:13.331699  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.332284  8697 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:13.334519 15068 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.289109 14733 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.289191 14688 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:13.302059 15019 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:13.352263  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:13.353617  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 73164.7us
W20260709 13:04:13.354178  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.290804 14962 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:13.357281  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 111874us
W20260709 13:04:13.361896  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.357249  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:13.344203 14733 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:13.336241  9325 rpcz_store.cc:276] 0709 13:04:12.151149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.152700 (+  1551us) service_pool.cc:224] Handling call
0709 13:04:12.153094 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:12.155545 (+  2451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.155591 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:12.156097 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 13:04:12.448505 (+292408us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b1800362da0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:12.740589 (+292084us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:12.748050 (+  7461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.767993 (+ 19943us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:13.300727 (+532734us) raft_consensus.cc:1685] finished
0709 13:04:13.300746 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.325944 (+ 25198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6216,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":24775936,"thread_start_us":259773,"threads_started":1,"wal-append.queue_time_us":917935}
I20260709 13:04:13.364710 14821 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.370880 14821 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.371827 15010 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:13.372519 15010 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:13.373644  9323 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:13.373962  9322 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 13:04:13.375906  9322 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 13:04:13.376381  9322 pending_rounds.cc:85] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 10
I20260709 13:04:13.376823  9322 pending_rounds.cc:107] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 13:04:13.377522  9322 raft_consensus.cc:2889] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:13.378448 14980 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.342469 14770 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.373734 15010 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.383471  9329 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:13.384564  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.385871 15019 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.389787  8724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:13.390372  8724 raft_consensus.cc:606] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:04:13.390694  8724 raft_consensus.cc:614] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:13.392059 14898 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.393755  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:13.394757  8577 raft_consensus.cc:2393] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:13.396358  8575 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:13.397790  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:13.398523  8582 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:13.401921  8724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:13.402628  8724 raft_consensus.cc:606] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.403093  8724 raft_consensus.cc:614] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:13.398082  9327 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:13.353349 15122 raft_consensus.cc:2764] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 13:04:13.407306  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:13.397945  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68582us
I20260709 13:04:13.408838  9330 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:13.355913 15097 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:04:13.406762 15020 consensus_meta.cc:287] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.659s	user 0.012s	sys 0.000s
W20260709 13:04:13.437242  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52800.4us
I20260709 13:04:13.433120 14688 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.439404 14962 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:13.440459  8697 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.435829  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:13.445081  9322 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:13.449882  8724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d851286f2cf248549fc81f8296784c08"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:13.448755 14267 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.450646  8724 raft_consensus.cc:606] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.451086  8724 raft_consensus.cc:614] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:13.409538 15044 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.462442  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.441293 15019 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.446018 14821 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.489159  8724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:13.409688 15101 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.490986  8724 raft_consensus.cc:606] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.491470  8724 raft_consensus.cc:614] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:13.471087 14267 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:13.455871 15003 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.465718 15044 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.495807  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:13.498008  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:13.464152 14770 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:13.502534  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 326651us
I20260709 13:04:13.487037 14962 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:13.448194 15020 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:13.510820  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 77489us
I20260709 13:04:13.514926  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.520603 14267 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:13.491639 14948 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:13.526965  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:13.538472  8658 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 13:04:13.542398  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:13.516377 14893 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:13.511644 14962 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:13.525374 15102 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.510222 15021 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:04:13.564587  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 119194us
W20260709 13:04:13.565186  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.493808 14821 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:04:13.541906 14770 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:04:13.577382  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62541.6us
I20260709 13:04:13.560679  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:04:13.552229 14962 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:13.517540 15044 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.548889 14893 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.595932  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:13.453919  8474 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 13, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 13 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.599768  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:13.537222 15003 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.576810 14770 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:13.613251  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:04:13.614045 15021 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:13.614418  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:13.615358  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:13.595191 15071 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:13.616029  9324 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:13.616375 15050 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:13.545804 14948 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:13.623010  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 86720.5us
I20260709 13:04:13.610270  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:13.596423 14893 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:13.630034  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:13.630728  8653 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:13.616029 14898 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 13:04:13.595709 14980 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:13.617149 15050 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.619354 14770 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:13.647992 14267 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:13.651646  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:13.608835 15103 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.658631 15044 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:13.661806  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70543.4us
I20260709 13:04:13.662693  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:13.663614 15010 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:13.664054  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.664156 15010 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.664780 15010 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:13.666368  8653 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:13.666839 15003 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.669154 14980 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.670215 15101 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.671291 14980 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.671670  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:13.672530 14906 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:13.672914 14906 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.673275  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:13.669512  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 73679.4us
I20260709 13:04:13.674325 14267 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.676404  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:13.678309  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:13.690668  9322 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:13.673259 14906 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:13.677736 15060 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.719s	user 0.017s	sys 0.000s
I20260709 13:04:13.674407 15097 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:13.696164  8656 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:13.646828 15071 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:13.692502  8623 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:13.695284 15137 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:13.677616 15021 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:13.706683 14505 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:13.721606  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:13.725769  8730 raft_consensus.cc:2410] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 17 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:13.726963  8548 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:13.710157 15097 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:13.730199  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:13.740069  8729 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:13.722513  8654 raft_consensus.cc:2376] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18. Re-sending same reply.
I20260709 13:04:13.707700 15137 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:13.744324  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:13.696975 14980 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 13:04:13.746734 14267 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.747479 14267 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:13.716012 14898 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:13.750094  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:13.751332  9322 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:13.753249  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:13.718964 14505 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.717604 15021 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:13.689455 15103 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.731720 15020 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.760807 14267 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:13.741154 15058 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.744553 15097 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:13.783563 15146 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.212s	user 0.000s	sys 0.002s
I20260709 13:04:13.762856 14505 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:13.787495 15136 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.172s
I20260709 13:04:13.770342  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:13.766049 15021 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:13.775559  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:13.777318 15050 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.777565 14765 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:13.779451  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:13.710462 15060 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.731215 14827 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 13:04:13.769526  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:13.818272  8732 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:13.746333  8474 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 9 to 10, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 10 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.819422  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:13.820536  8651 raft_consensus.cc:2393] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:13.821092  8474 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 12, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:13.826411  8474 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 13, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 13 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:13.818305 14827 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:13.830065 14827 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:13.831085 14827 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.832437 14827 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:13.832342 15010 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:13.793823  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 87519us
I20260709 13:04:13.795080  8725 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:13.720935 15071 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.830013 15103 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:13.835539 15071 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.836369  8651 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:13.835884 15137 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.837432 15137 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:13.838507 15060 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.840459 14948 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.841958 14980 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:13.842648 14980 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.844735  8729 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:13.768184 14898 raft_consensus.cc:697] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:13.862135  8655 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:13.870112 14267 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:13.814971 14765 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.828389  8474 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 9 to 10, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 10 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.890070  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:13.891606  8653 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:13.896368  8659 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:13.906450  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:13.920280  8575 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 13:04:13.890488 14765 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:13.925714  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:13.926620  8582 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:13.926914  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:13.929261  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.931772  9327 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:13.913900 14948 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:13.942395  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 269154us
I20260709 13:04:13.948748  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:13.950546  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:13.950982  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:13.951748  8578 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
I20260709 13:04:13.952579 14947 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:13.953065 14947 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.956009  8726 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:04:13.957253  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.959091 14980 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:13.961169 15010 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:13.942266 15102 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
W20260709 13:04:13.968443 14733 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.533s	user 0.006s	sys 0.002s
I20260709 13:04:13.874678 15058 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:13.986586  8657 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:04:13.936231 15122 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:13.988332 14947 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:14.027134  8698 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:14.031318  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.032054  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:14.035243 14267 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:04:14.032595  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 127320us
I20260709 13:04:14.036952  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:14.051146  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:14.052238  9327 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:04:14.054383 14267 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:14.003724 14733 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.068692  8732 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 13:04:14.070675  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:14.073751  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 365574us
I20260709 13:04:14.080211  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:14.080442  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54585.4us
I20260709 13:04:14.043392 15010 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:14.087440  8724 raft_consensus.cc:2393] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:14.088445  9328 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:14.027866 14505 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.030212 14948 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.092223 14267 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:14.106246  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 271255us
I20260709 13:04:14.109118  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:14.121768  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.031363 14898 consensus_queue.cc:237] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.125826  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:14.127086  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:14.128422  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:14.128847  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.128993  8655 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:14.130134  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:14.117545 15071 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:14.144388 15021 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:14.145846  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 65713.7us
I20260709 13:04:14.081372  8474 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:14.149580 15165 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.120s
I20260709 13:04:14.097194 15010 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:14.152626 15010 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.076355 15060 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 13:04:14.148520  8474 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 5 to 13, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 13 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:14.204109  8659 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:14.130883  8651 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:14.214960  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
W20260709 13:04:14.155368 15153 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.562s	user 0.000s	sys 0.005s
W20260709 13:04:14.224344 15169 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:04:14.224759  8575 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:14.225593 15101 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.226224 15101 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.226953  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3276) took 1331 ms. Trace:
I20260709 13:04:14.228353 15097 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.218832  8577 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:14.231259  8729 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:14.232663  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1412) took 1165 ms. Trace:
I20260709 13:04:14.234393 15083 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:14.235433 15101 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:14.238178  8474 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:14.240217 14893 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.002s
I20260709 13:04:14.240706 14893 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.210558  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.222326 14948 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:14.222532 15137 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:14.223119 15154 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.702s	user 0.004s	sys 0.015s
I20260709 13:04:14.175375 14733 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:14.223860 14505 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:14.227360  8652 rpcz_store.cc:276] 0709 13:04:12.895380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.898707 (+  3327us) service_pool.cc:224] Handling call
0709 13:04:12.899196 (+   489us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:12.899861 (+   665us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.899898 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:12.900722 (+   824us) log.cc:844] Serialized 52 byte log entry
0709 13:04:13.281455 (+380733us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:13.344078 (+ 62623us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.366821 (+ 22743us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.429566 (+ 62745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.429681 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:13.534854 (+105173us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b1800507e20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:04:14.154668 (+619814us) spinlock_profiling.cc:243] Waited 8.8 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:14.226685 (+ 72017us) raft_consensus.cc:1685] finished
0709 13:04:14.226711 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.226869 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110844,"prepare.queue_time_us":48316,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":19149,"raft.queue_time_us":112359,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":62729,"spinlock_wait_cycles":32092800,"thread_start_us":380130,"threads_started":1,"wal-append.queue_time_us":604402}
W20260709 13:04:14.215629 14770 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.520s	user 0.006s	sys 0.006s
I20260709 13:04:14.106072 15130 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.244119  8655 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:14.260483  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:14.267436  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:14.268496  8577 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:14.242260  9327 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:14.270756  9327 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:14.270234  8584 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:14.257774 15041 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:14.280654  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.282972  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.282260  8625 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:14.256945 15060 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.233134  9326 rpcz_store.cc:276] 0709 13:04:13.066705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.066855 (+   150us) service_pool.cc:224] Handling call
0709 13:04:13.109338 (+ 42483us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:13.315684 (+206346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.315720 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:13.316448 (+   728us) log.cc:844] Serialized 52 byte log entry
0709 13:04:13.599112 (+282664us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:14.072128 (+473016us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.072659 (+   531us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.231801 (+159142us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b18001578a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f14097ccd70 00007f14097c0293 00007f14097b5cea 00007f14097b623a 00007f1409c20a2a 00007f1409c300bf 00007f1409c30038
0709 13:04:14.232120 (+   319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.232198 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:14.232333 (+   135us) raft_consensus.cc:1685] finished
0709 13:04:14.232347 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.232466 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94211,"prepare.queue_time_us":637713,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":91,"raft.queue_time_us":41500,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":12825,"spinlock_wait_cycles":9985280,"thread_start_us":692696,"threads_started":2,"wal-append.queue_time_us":454359}
I20260709 13:04:14.215906 14906 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:14.330078  8576 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.014s	sys 0.017s
W20260709 13:04:14.344231  8576 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s	user 0.014s	sys 0.017s
W20260709 13:04:14.347757  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 65560.1us
W20260709 13:04:14.347733  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 309340us
I20260709 13:04:14.356083  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:14.358258  8575 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:14.265923 15137 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.294230 15050 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:14.348708  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:14.342324  9327 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.361238 14948 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:14.264191 14770 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:14.364061  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 338112us
I20260709 13:04:14.241508 14765 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:14.371977  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.389829  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:14.396883  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:14.281028 15170 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:14.295258 15130 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.421350  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:14.424463  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:14.430766  8651 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:14.431852  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:14.430789  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:14.432793  9330 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:14.411446  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.432797  8583 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:14.436151  9327 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:14.384869 15104 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:14.334261 15060 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.438489  9323 raft_consensus.cc:2410] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:14.442581  8584 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:14.446715  8577 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:14.443346  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:14.405287 15167 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:14.398242 15165 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 13:04:14.406629 15020 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.046s
W20260709 13:04:14.467592  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 332434us
I20260709 13:04:14.472287  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:14.442060 15104 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.469648  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.441465 15136 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:14.241199  8474 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 2 to 12, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:14.477047  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.455819 15097 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:14.461760 15021 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:14.506421  8581 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:14.524621  8474 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 12, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 12 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:14.536075 15137 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:14.480244 15104 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:14.537014  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.537815  8584 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:14.538766  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.539635 14765 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:14.480932 14962 consensus_meta.cc:287] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.873s	user 0.013s	sys 0.000s
I20260709 13:04:14.508912  8580 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:14.471984 15167 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:14.479059 14821 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.569523  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:14.570609 15003 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:14.571883 15003 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:14.505268 14963 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:14.576546  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:14.584087 14267 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:14.584699 14267 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:14.594596  8575 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:14.594521  8584 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:14.596150  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:14.612159  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 212326us
W20260709 13:04:14.614095  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.618006  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:14.618865  8580 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:14.620338  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.612003  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:14.622774  8581 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
W20260709 13:04:14.632185  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 159953us
I20260709 13:04:14.633754  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:14.598788 14947 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.544s	user 0.012s	sys 0.008s
W20260709 13:04:14.640552  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.624507 14267 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.635684  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:14.641930  9329 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:14.573314 15003 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.655416  9327 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:04:14.509455 15136 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.580623 14963 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:14.616345 14893 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:14.629653 15097 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:14.645944  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 197016us
I20260709 13:04:14.669189  8584 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:14.674752  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.575739 14962 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:14.670497  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 133949us
I20260709 13:04:14.594983 14821 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.677063  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:14.680256  8582 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:14.682753  8581 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 13:04:14.684945  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 51278.2us
I20260709 13:04:14.687454  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:14.673318 14963 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:14.526744  8474 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:14.748292  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 112626us
W20260709 13:04:14.749100  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 61745.5us
I20260709 13:04:14.749922  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:14.679632 15071 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:14.651741 14947 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:14.757899 15194 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260709 13:04:14.759744  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 145337us
I20260709 13:04:14.689137 14906 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:14.771991  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 287755us
I20260709 13:04:14.698276 14505 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:14.778538  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.787268 15071 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.790334 15097 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:14.790014 15182 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:14.769873 15167 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.689851 15020 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260709 13:04:14.792399 14947 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:14.806672  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45370.9us
I20260709 13:04:14.807873  9329 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
W20260709 13:04:14.809509  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 59671.4us
I20260709 13:04:14.810622  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:14.759372 14948 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:14.794525 15214 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:14.817667  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:14.824118  8579 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 13:04:14.814062 15183 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.398s	user 0.001s	sys 0.001s
I20260709 13:04:14.828003  8578 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:04:14.805125 14770 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:14.816531 15179 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
W20260709 13:04:14.817719 15187 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.381s	user 0.004s	sys 0.009s
I20260709 13:04:14.839432  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:14.840234  8659 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:14.841172  8652 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:14.816403 15130 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:14.830250  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:14.818655 15097 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:14.833673  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.811209 15040 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:14.826115 15196 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.057s	user 0.000s	sys 0.003s
I20260709 13:04:14.861756  8581 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:14.861821 15040 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.830231  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:14.843734  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:14.819824 15214 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.864982 15214 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:14.865537  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:14.867165 15130 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:14.855439  8582 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:14.867542  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:14.810129 14505 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:14.868336 15130 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:14.870136 15186 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:04:14.870362 14505 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:14.871202  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:14.872136  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1097) took 1020 ms. Trace:
I20260709 13:04:14.856561  8657 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:14.857926  8579 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:04:14.873926  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1390) took 1031 ms. Trace:
I20260709 13:04:14.875236  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:14.830662 14898 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:14.878700  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:14.880198  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:14.880772  8732 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:14.819859 15137 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.028s
I20260709 13:04:14.874367  9322 rpcz_store.cc:276] 0709 13:04:13.842488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.842651 (+   163us) service_pool.cc:224] Handling call
0709 13:04:13.842980 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:13.843348 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.843378 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:13.843855 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 13:04:14.085475 (+241620us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:14.085523 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.085616 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:14.873708 (+788092us) raft_consensus.cc:1685] finished
0709 13:04:14.873731 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.873866 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2082,"raft.queue_time_us":6961,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":10419,"thread_start_us":232286,"threads_started":1,"wal-append.queue_time_us":448410}
I20260709 13:04:14.872560  8658 rpcz_store.cc:276] 0709 13:04:13.852222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.852402 (+   180us) service_pool.cc:224] Handling call
0709 13:04:13.890946 (+ 38544us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:13.903043 (+ 12097us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.903096 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:13.918797 (+ 15701us) log.cc:844] Serialized 52 byte log entry
0709 13:04:14.044492 (+125695us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:14.104513 (+ 60021us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.166849 (+ 62336us) spinlock_profiling.cc:243] Waited 879 us on lock 0x7b180024cc40. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:04:14.167106 (+   257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.167179 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:14.871863 (+704684us) raft_consensus.cc:1685] finished
0709 13:04:14.871971 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.872088 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1394,"prepare.queue_time_us":56899,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":12402,"raft.queue_time_us":62835,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":13222,"spinlock_wait_cycles":2022784,"thread_start_us":124989,"threads_started":1,"wal-append.queue_time_us":764929}
I20260709 13:04:14.860749 14821 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:14.901286  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.860985 15097 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:14.903234 15097 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:14.821585 15203 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 13:04:14.822044 14948 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.901286  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:14.919466  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:14.913461  9330 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:14.914834  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:14.920869  8724 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:14.858826 15204 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:14.876225 14980 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.879846 14770 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.924392  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.883576 15021 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.887691 14505 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:14.930166  8730 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:14.931882  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:14.872836  8474 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 9 to 11, leader changed from <none> to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67), now has a pending config: VOTER a0d703ee18464475a290425976246555 (127.7.78.65), VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66), VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67), VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 11 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } } pending_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:14.910000 15101 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:14.938849  8658 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:14.921751 15212 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:14.944839  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.948117  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:14.951922  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:14.944927 15204 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:14.955209 15204 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.957229 15136 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:14.957732  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:14.958505  8726 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
W20260709 13:04:14.958568 15197 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 13:04:14.959290  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1094) took 1207 ms. Trace:
I20260709 13:04:14.952569 15101 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.959764  8654 rpcz_store.cc:276] 0709 13:04:13.751923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.770785 (+ 18862us) service_pool.cc:224] Handling call
0709 13:04:14.959080 (+1188295us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b540005a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:14.959226 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2023225088}
I20260709 13:04:14.932446 14770 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:14.962021 15208 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:04:14.963498 14770 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:14.949527 15212 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:14.971342 15212 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:14.972535 15177 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 13:04:14.972762 15212 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:14.941793 15021 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:14.974040  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:14.982923  8578 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
W20260709 13:04:14.988878  8651 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.014s
I20260709 13:04:14.989337  8651 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:15.005915  8730 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:04:15.009280  8730 pending_rounds.cc:85] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 7
I20260709 13:04:14.935719 14505 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:15.011041  8730 pending_rounds.cc:107] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 13.8
W20260709 13:04:15.012068  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.013693  8730 raft_consensus.cc:2889] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:14.953532 15188 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 13:04:15.013561  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.016294  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:15.017093  8726 raft_consensus.cc:2368] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:15.019586  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.020340  8579 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.023325  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.023386  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:15.025575  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.930429 14980 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.027936  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.037659  8575 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:15.029958  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.008272 15213 raft_consensus.cc:2764] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:04:15.016032 14505 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:15.014763 14770 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:15.040678  8581 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:15.041730  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.037528  8580 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:14.982622 14947 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:15.051669 14980 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.055550  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:15.049312  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.062884 15058 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:15.064095  8583 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.621s	user 0.018s	sys 0.000s
I20260709 13:04:15.069662  8583 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
W20260709 13:04:15.071501 15181 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.626s	user 0.003s	sys 0.000s
I20260709 13:04:15.071693 14947 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.075269 14947 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:15.072857 15199 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.078795  8726 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:15.079589  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1414) took 1817 ms. Trace:
I20260709 13:04:15.077217 15199 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.083410  8724 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:15.085662  8730 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:15.086647  8580 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:15.080127  9331 rpcz_store.cc:276] 0709 13:04:13.262969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.361235 (+ 98266us) service_pool.cc:224] Handling call
0709 13:04:13.435340 (+ 74105us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:13.733454 (+298114us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.733487 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:13.771366 (+ 37879us) log.cc:844] Serialized 52 byte log entry
0709 13:04:14.090641 (+319275us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:14.090718 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.090828 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.071069 (+980241us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b24000c5318. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:04:15.078241 (+  7172us) raft_consensus.cc:1685] finished
0709 13:04:15.078263 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.079513 (+  1250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4844,"raft.queue_time_us":41169,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":127,"spinlock_wait_cycles":38074752,"thread_start_us":537337,"threads_started":2,"wal-append.queue_time_us":515050}
I20260709 13:04:15.102994  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.105566  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:15.072105 15207 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.108196  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.108619  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.155932  8580 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:15.016196 15189 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 13:04:15.138002  8732 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:15.139819  8730 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
W20260709 13:04:15.144793  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 76262.1us
I20260709 13:04:15.106626 14770 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.139072 15021 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.159967  8583 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:15.166695  8724 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 13:04:15.167057  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.164430  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.166567  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.167855  8580 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:15.168110  8583 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:15.168779 14267 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.155900  8575 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:15.169250 14267 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.182971  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.184540  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.185824  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.186831  8584 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:15.158540 14948 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.188165  8726 raft_consensus.cc:2393] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:15.191035  8581 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.183882  8578 raft_consensus.cc:2376] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17. Re-sending same reply.
I20260709 13:04:15.193066  8580 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:15.195961  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:15.195199  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.195943  8658 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:04:15.212823  8655 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:15.201570  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.219554  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.221733 15122 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:15.222095 14963 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.222333 15122 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.224601 15122 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.201695  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.183250 15003 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:15.231235  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.235481 15003 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.232297  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.228842  8576 raft_consensus.cc:2376] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18. Re-sending same reply.
I20260709 13:04:15.188549 14898 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:15.198405 15170 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:15.248617 15201 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.460s	user 0.001s	sys 0.000s
I20260709 13:04:15.253521  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.253521  8726 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:04:15.270126 14267 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:15.271260  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.256729  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.260242 14980 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:15.267395 14962 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.255353 15199 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.283959  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:15.285346  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:15.286496  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:15.271253  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:15.255931  8698 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:15.273403  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:15.238802 15219 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:04:15.217387 15101 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.275628  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.276417  8652 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:15.278575  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.278591 15203 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.291487  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.336038  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.258113 15003 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.337270  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:15.282861 15041 consensus_meta.cc:287] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.990s	user 0.004s	sys 0.019s
I20260709 13:04:15.340726  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:15.165294 15207 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.345909  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:15.270907 15170 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:15.355257  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:15.355607  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.357443  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.362525  8659 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:04:15.357800  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:15.358815  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 102982us
I20260709 13:04:15.359913 14267 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:15.371248  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:15.278918 15103 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 13:04:15.374181  8650 raft_consensus.cc:2368] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16.
I20260709 13:04:15.333803 14893 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:04:15.374913  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 129969us
I20260709 13:04:15.333760 15167 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:04:15.333963 15130 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:15.379565  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:15.278820 15210 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.103s
I20260709 13:04:15.290342 15214 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.279410 14898 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.336197 15203 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.354966 15199 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.104s
I20260709 13:04:15.410502  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.459450  8659 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:15.460528  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:15.398947 14961 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 13:04:15.340101 14505 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.462414  8651 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:15.380579 14893 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:15.463173 14893 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:15.464373 14505 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.381199  8550 leader_election.cc:400] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:15.465232  8550 leader_election.cc:403] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
W20260709 13:04:15.447423 15225 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 13:04:15.465767  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 127901us
I20260709 13:04:15.410847  8650 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:15.468072  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.360673 15003 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:15.361490 15041 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:15.469913 15101 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:15.385185  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.470889 15210 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:15.471381 15210 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:15.471453  8726 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 16 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 13:04:15.472741 15097 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.569s	user 0.008s	sys 0.001s
I20260709 13:04:15.472445 15210 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.473563  8582 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:15.474015  9331 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:15.474381  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:15.474630  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:15.385151  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.473874  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:15.475804  9330 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:04:15.429659 15200 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:15.477435  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:15.399019  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43869.5us
I20260709 13:04:15.483568 15210 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:15.417713  8579 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:15.347002 14821 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.480010  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:15.487637  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:15.488039 14821 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.488478 14948 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:15.489012 14948 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:15.490015 14948 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.490623 14821 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.409389 15058 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:15.473171 15097 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.491861  8725 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:15.493724 14821 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:15.483927  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:15.384718  8655 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:04:15.495090 15191 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:15.495539 15191 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:15.496426 15191 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.497004 15200 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:15.484340  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:15.497279 14821 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:15.497890  9330 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:15.485059  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:15.485556  9325 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:04:15.500301 15191 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:15.501935  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.482800  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:15.506062  9326 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:15.506959  8579 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:15.505015 14962 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:04:15.505508  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.504822 15200 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:15.507277  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:15.469213  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:15.518323  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:15.518957  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.477705 15083 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.529845  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:15.530287 15083 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.531119 15190 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:15.531854 15190 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:15.532516 15083 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.531275 14962 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:15.533053 14962 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:15.534107  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:15.551671  8730 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:15.454097 15223 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 13:04:15.547253 14770 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.381338 15173 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:15.470659 15101 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 13:04:15.561288  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:15.586961  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:15.576066  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71432.9us
W20260709 13:04:15.410655 15104 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.873s	user 0.006s	sys 0.014s
I20260709 13:04:15.577229  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.597432  8576 raft_consensus.cc:2376] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17. Re-sending same reply.
I20260709 13:04:15.580917  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:15.599685  9322 raft_consensus.cc:2368] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15.
I20260709 13:04:15.570241 15170 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.585589  8581 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:15.613473  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:15.588522  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:15.458077 15227 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:04:15.593830  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.618034  8575 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:15.606417  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.619086  8579 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:15.607753  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.620361  8583 raft_consensus.cc:2376] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17. Re-sending same reply.
I20260709 13:04:15.608498  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:15.612075 15203 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.354602 15136 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.219s
I20260709 13:04:15.501081 14893 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.615414  8657 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:15.616796  8696 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:15.614365  8655 raft_consensus.cc:2368] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15.
W20260709 13:04:15.622751  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 155999us
W20260709 13:04:15.613443  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 242260us
I20260709 13:04:15.585484 14267 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:15.573490 15103 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:15.635653  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:15.589035 14906 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:15.638789  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.642751  8650 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:15.642869  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 208988us
I20260709 13:04:15.617820 15210 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:04:15.632189 14947 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.556s	user 0.010s	sys 0.001s
I20260709 13:04:15.645303  8654 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:15.640076  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:15.648658  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.647545  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:15.654316  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.655251  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:15.647496 15165 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:15.652215 15210 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.640617 14906 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:15.614893 15104 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.742765  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:15.645792 14962 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.656661 14947 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.823858 15165 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:15.780617  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 127376us
I20260709 13:04:15.848392 15165 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.848896  8698 leader_election.cc:400] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16.
W20260709 13:04:15.781363  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.809885 15097 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.849267  8698 leader_election.cc:403] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:04:15.852069  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.853118  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:15.853921 15130 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:15.854311 15130 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.810101  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.854681 15130 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:15.810496  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.814465 14898 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.855248  8650 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:15.856616  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:15.856851 14962 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.857286 15040 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.533041 15190 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:15.652886 14267 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.859203  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:15.813200 15212 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.840s	user 0.018s	sys 0.012s
I20260709 13:04:15.816874 15170 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.816411 15200 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:15.631770 15058 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 13:04:15.653476 15103 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:15.639789 15021 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:15.659063 15249 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:15.650239 15173 raft_consensus.cc:2804] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:15.815588 15204 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:15.857452 14898 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:15.860546  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:15.821913  8622 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:15.879101  8658 raft_consensus.cc:2368] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19.
I20260709 13:04:15.873018 15200 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.827432 15210 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:04:15.884855  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.866264 15097 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:15.892536  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82895.6us
I20260709 13:04:15.862133 14906 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.886551 15040 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 13:04:15.931371  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:15.932155  8582 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:15.932439 15245 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:15.933478 15250 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:15.933991 15250 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.934342 15250 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.896100 15137 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:04:15.926671 15212 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:15.877501 14770 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:15.935866  8733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2846) took 2278 ms. Trace:
I20260709 13:04:15.935949 14770 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.936364  8733 rpcz_store.cc:276] 0709 13:04:13.657331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.659175 (+  1844us) service_pool.cc:224] Handling call
0709 13:04:13.659495 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.861011 (+201516us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:14.331523 (+470512us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.670222 (+338699us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:14.670301 (+    79us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:14.677018 (+  6717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.253360 (+576342us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b18004d9620. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:04:15.935480 (+682120us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400436f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:15.935530 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.935665 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56413,"prepare.queue_time_us":343800,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":155602,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"spinlock_wait_cycles":433287296,"thread_start_us":674552,"threads_started":2,"wal-append.queue_time_us":427624,"wal-append.run_cpu_time_us":1357,"wal-append.run_wall_time_us":1049479}
I20260709 13:04:15.938352 14770 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.939875 15253 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.933251 15130 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:15.940204  8582 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:15.940423 15253 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:15.892557  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 79907.7us
I20260709 13:04:15.876677 15214 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.896252  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:15.880865 15170 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16.
I20260709 13:04:15.881884 15207 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:15.941993 14898 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:15.941123 15200 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:15.941998 15130 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:15.975533  8575 raft_consensus.cc:2376] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15. Re-sending same reply.
I20260709 13:04:16.033558  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.047345  8724 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:16.036635  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:16.035851  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 173239us
I20260709 13:04:15.980372  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:15.988973  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2864) took 1073 ms. Trace:
W20260709 13:04:16.052048  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71803.5us
I20260709 13:04:16.051797  8725 rpcz_store.cc:276] 0709 13:04:14.915251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.915359 (+   108us) service_pool.cc:224] Handling call
0709 13:04:14.924154 (+  8795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:15.492293 (+568139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.943177 (+450884us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b5400416754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:15.988893 (+ 45716us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1305108224}
I20260709 13:04:15.945407  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:16.057644  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:15.893234 15021 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:16.061008  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 115681us
W20260709 13:04:16.062633  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.063077  8724 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 13:04:16.063459  8724 pending_rounds.cc:85] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:04:16.063764  8724 pending_rounds.cc:107] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 13:04:16.064067  8724 raft_consensus.cc:2889] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:16.064025  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.035167 14688 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.098349  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.088670 15130 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.989457 15173 raft_consensus.cc:697] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:15.989498 15212 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:15.997534 15097 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:16.180135  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 81896.4us
I20260709 13:04:16.218045  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.053954 14770 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.118441  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2639) took 1839 ms. Trace:
I20260709 13:04:16.103873  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:16.059585 15167 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:16.121817  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:16.234726  8577 rpcz_store.cc:276] 0709 13:04:14.279488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.283054 (+  3566us) service_pool.cc:224] Handling call
0709 13:04:14.321462 (+ 38408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.447317 (+125855us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.118201 (+1670884us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54003449d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:16.118360 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287567616}
W20260709 13:04:16.180135  9297 connection.cc:256] RPC call timeout handler was delayed by -1.3936780679999856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
I20260709 13:04:16.104537 14267 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.181874  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:16.103717 15233 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.200s	user 0.008s	sys 0.007s
I20260709 13:04:16.112179  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.251531  8725 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:16.050798 14893 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.176438 15058 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.250144  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.254436  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.255270  8731 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:16.107210 15244 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.113212 15060 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:16.112378 15248 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:16.269942 15248 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.271466 15097 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.181722 14505 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:16.272135 14267 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.272534 14505 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.273308  8733 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:16.259166  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.256367 15261 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:16.275450  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:16.275565 15261 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:16.276983 15261 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:16.258544 15058 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.277925 15261 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:16.278530 15261 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.278991 15261 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.281008 14906 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.181378 15260 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:16.286705  8726 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:16.101847 14688 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.264197  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:16.274298 14267 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.285859 15256 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:16.061434 15207 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.169s
I20260709 13:04:16.105119 15130 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.250893 15167 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.304212  8724 raft_consensus.cc:2410] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:04:16.288923 15260 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.272159 15248 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:16.328192  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 305176us
I20260709 13:04:16.284569 15261 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:16.341737  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.117899 15253 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.291755 15104 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.248996 14770 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:16.390580  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.391413  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.329488 15167 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:16.396322  8622 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.513s (SENT)
I20260709 13:04:16.288334 15165 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.330910 15260 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:16.117131 14898 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.291298 15244 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:16.396440 15233 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 13:04:16.398802  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.013s	sys 0.012s
I20260709 13:04:16.346822 14906 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:16.409060  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.014s	sys 0.013s
I20260709 13:04:16.416555  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.418308  8577 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:16.420899  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.422273  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 204316us
I20260709 13:04:16.427415  8549 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.395172 15212 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.439260  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:16.455116  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.421043 15262 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:16.459313  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.466205 15262 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.471969  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.472744  8577 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:16.472822 15261 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:16.474185 15244 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:16.474337 15263 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:04:16.471449 15260 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.394781 15173 consensus_queue.cc:237] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.483342 15260 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.502385 15130 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.503849  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1227) took 1479 ms. Trace:
I20260709 13:04:16.504829 15068 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:16.511090 15068 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.512132 15102 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:16.508846 15236 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.050s	user 0.004s	sys 0.009s
I20260709 13:04:16.512840 15102 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.504290  8729 rpcz_store.cc:276] 0709 13:04:15.024640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.037247 (+ 12607us) service_pool.cc:224] Handling call
0709 13:04:15.048000 (+ 10753us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:15.862316 (+814316us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:15.881577 (+ 19261us) log.cc:844] Serialized 14 byte log entry
0709 13:04:16.176119 (+294542us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:16.217753 (+ 41634us) log.cc:844] Serialized 26 byte log entry
0709 13:04:16.217965 (+   212us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:16.218014 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.503417 (+285403us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b5400472cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:16.503468 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.503521 (+    53us) raft_consensus.cc:1685] finished
0709 13:04:16.503635 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.503787 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1754,"prepare.queue_time_us":12412,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":60025,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":7851,"spinlock_wait_cycles":807676800,"thread_start_us":699397,"threads_started":2,"wal-append.queue_time_us":497417}
I20260709 13:04:16.517495  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1246) took 1318 ms. Trace:
I20260709 13:04:16.517730 14770 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.518625 15102 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.517730  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.517922  8578 rpcz_store.cc:276] 0709 13:04:15.198959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.199784 (+   825us) service_pool.cc:224] Handling call
0709 13:04:15.234169 (+ 34385us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:15.241568 (+  7399us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:04:15.313033 (+ 71465us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.322716 (+  9683us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:04:15.363747 (+ 41031us) log.cc:844] Serialized 78 byte log entry
0709 13:04:15.363962 (+   215us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:15.364228 (+   266us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.385906 (+ 21678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.386014 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.508238 (+1122224us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:16.517248 (+  9010us) raft_consensus.cc:1685] finished
0709 13:04:16.517272 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.517437 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46101,"prepare.queue_time_us":162863,"prepare.run_cpu_time_us":120,"prepare.run_wall_time_us":6807,"raft.queue_time_us":88029,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":27471,"spinlock_wait_cycles":110001280,"thread_start_us":65135,"threads_started":3,"wal-append.queue_time_us":47233}
I20260709 13:04:16.521832  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.522505  8729 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:16.523766  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:16.520408 14963 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:16.527426 15165 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.433274 15180 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.095s
I20260709 13:04:16.528502 15068 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.528044  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:16.531047 15180 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:16.531518 15180 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.531953 15180 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:16.460700 14688 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:16.540025  8657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.039s	sys 0.048s
W20260709 13:04:16.540637  8657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.039s	sys 0.048s
I20260709 13:04:16.536566  8729 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260709 13:04:16.498625 15021 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.508301 15200 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:16.543291  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 207373us
I20260709 13:04:16.458184 15256 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:16.551502  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.552101  8728 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 13:04:16.552281  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 304697us
W20260709 13:04:16.555354  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 100325us
I20260709 13:04:16.556077  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:16.462809 15248 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:16.564293  8725 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:16.546438 15222 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
W20260709 13:04:16.570757 15210 consensus_meta.cc:287] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.002s
W20260709 13:04:16.554769 15250 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.620s	user 0.009s	sys 0.002s
I20260709 13:04:16.563721  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.571209 15210 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:16.614917  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51259.7us
I20260709 13:04:16.616536  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.557499 15256 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:16.580070  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:16.621881  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.571899 15182 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:16.632086  8622 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.551s (SENT)
W20260709 13:04:16.633033  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 55501.2us
I20260709 13:04:16.631614  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:16.638458  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58502.7us
I20260709 13:04:16.618330 15207 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.644371  8729 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:04:16.627852 14688 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.647115  9326 raft_consensus.cc:1275] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:16.624792 15274 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:16.652657  8549 leader_election.cc:400] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15.
I20260709 13:04:16.653177  8549 leader_election.cc:403] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 13:04:16.562197 15273 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 13:04:16.628163 15101 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:16.651006  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:16.654924  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:16.657342  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:16.631032 15020 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:16.628101 15068 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:16.577637 15237 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:04:16.662181  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:16.631520 15182 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:16.658263  9323 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:16.665329  9294 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:16.585842 15250 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.679461  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.668213  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.642014 15122 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:16.662978 15020 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:16.684094 14893 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:16.668241  9326 raft_consensus.cc:2376] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15. Re-sending same reply.
I20260709 13:04:16.678519 14765 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:16.657140 15021 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:04:16.682683 15267 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:04:16.685518 15021 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:16.686349  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:16.686946 15269 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:16.684506 15020 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:16.687930  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:16.688114  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2662) took 1097 ms. Trace:
I20260709 13:04:16.685060 14765 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:16.683849  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.668424 15165 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:16.691254  8584 rpcz_store.cc:276] 0709 13:04:15.590870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.039510 (+448640us) service_pool.cc:224] Handling call
0709 13:04:16.039980 (+   470us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:16.040507 (+   527us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:16.050195 (+  9688us) log.cc:844] Serialized 14 byte log entry
0709 13:04:16.332814 (+282619us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:16.397381 (+ 64567us) log.cc:844] Serialized 52 byte log entry
0709 13:04:16.397600 (+   219us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:16.401109 (+  3509us) log.cc:844] Serialized 14 byte log entry
0709 13:04:16.401663 (+   554us) log.cc:844] Serialized 14 byte log entry
0709 13:04:16.413529 (+ 11866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.413656 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.687903 (+274247us) raft_consensus.cc:1685] finished
0709 13:04:16.687925 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.688059 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12834,"prepare.queue_time_us":16736,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":1884,"raft.queue_time_us":11892,"raft.run_cpu_time_us":641,"raft.run_wall_time_us":13061,"thread_start_us":243769,"threads_started":1,"wal-append.queue_time_us":399498}
I20260709 13:04:16.700218  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:16.672183 15182 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:16.700577  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.710000  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.711576 15237 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:16.712220 15237 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:16.654632 14688 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.713994 14765 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.715168 15250 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.715413 14765 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:16.715829 15250 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:16.716383 15250 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.671113 15041 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15.
I20260709 13:04:16.670491 15222 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:04:16.678679 14963 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:16.718590  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46603.5us
I20260709 13:04:16.719468  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:16.656764 15207 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.721128  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.726771  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:16.734117  8583 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:04:16.744616  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.745287  8583 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:16.745515  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.746236  8575 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260709 13:04:16.737946  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66897.9us
I20260709 13:04:16.669512 15068 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.740633  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.708894  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:16.742935  8582 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:16.758175  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.759727  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.750572  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.762076  8577 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:16.763254  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.764642  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.761161  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.707643 15165 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.777837  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.717821 15269 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
I20260709 13:04:16.788573 15279 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:16.789441 15279 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:16.789516 15282 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 13:04:16.790012 15279 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.791779  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.792918  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.793257  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3313) took 1386 ms. Trace:
I20260709 13:04:16.793711  8654 rpcz_store.cc:276] 0709 13:04:15.407408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.434776 (+ 27368us) service_pool.cc:224] Handling call
0709 13:04:15.435194 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:15.651894 (+216700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.793010 (+1141116us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b5400139a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:16.793195 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":482715008}
I20260709 13:04:16.794665 15279 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.796525  8654 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:16.698819 15122 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.798676 15122 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.801349  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.802019  8581 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:16.803594 15136 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.805248 15200 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:16.806099 15165 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:16.807291  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.807737  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.808425  8659 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:04:16.707273 15101 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:16.829952  8583 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:16.809648  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:16.806133  8730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 3027) took 1551 ms (client timeout 1690 ms). Trace:
I20260709 13:04:16.804181  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:16.814677  8696 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:16.815011  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 135621us
I20260709 13:04:16.815969  8581 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:16.825668  8580 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:16.834175  9297 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.833240  8653 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:16.744972 15102 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:16.841545  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:16.832190  8730 rpcz_store.cc:269] 0709 13:04:15.254226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.325944 (+ 71718us) service_pool.cc:224] Handling call
0709 13:04:16.805908 (+1479964us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b540022a954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:16.806075 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":492163456}
I20260709 13:04:16.849035  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.849745  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.851393  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.856242  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.859923  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:16.857142  8725 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:16.810316 15041 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:16.866925 15189 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:16.828797 15137 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.878746  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.865535  8730 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:16.853682 15102 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 20 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:16.744848 14963 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.885752  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.886674  8578 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:04:16.855129 15103 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:16.863173 15275 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:16.822973 15173 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 13:04:16.872028 15200 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:16.845664 15278 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:04:16.855326 15253 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 13:04:16.889792 15103 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.897485 15200 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:16.906903 15103 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:16.907207 15200 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:16.908579 15294 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.063s	user 0.000s	sys 0.002s
I20260709 13:04:16.906051  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:16.913616  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.890086 15189 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:16.879526 15041 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.915904 15020 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.916002 15189 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.917205 15041 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.837311 15136 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.884622 15137 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:16.920708 15020 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:16.906479 15210 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:16.902482 15193 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:16.924571 15180 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:16.906492 15279 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:04:16.927291 15190 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:16.927635 15278 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:16.929140  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1230) took 1903 ms. Trace:
I20260709 13:04:16.939056  8732 rpcz_store.cc:276] 0709 13:04:15.025468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.109224 (+ 83756us) service_pool.cc:224] Handling call
0709 13:04:15.109534 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:15.161765 (+ 52231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.928898 (+1767133us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b5400416254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:16.929077 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65013120}
W20260709 13:04:16.931962  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.910089 15167 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.002s
I20260709 13:04:16.944559  8578 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:16.946045  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:16.875331 14748 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:04:16.940954  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:16.964399  8732 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
W20260709 13:04:16.965379  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59419us
I20260709 13:04:17.019814  8584 raft_consensus.cc:2368] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:04:17.019917  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:16.892529 15280 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
W20260709 13:04:17.030490  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 62176.8us
I20260709 13:04:16.969360 15275 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:17.041550  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.042372  8578 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:17.043699 15207 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.045028  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.014484 15137 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.002851 15102 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.056196 14947 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.058740 15180 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.041981 15290 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.061532 15290 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.062227 15058 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.063792 15290 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.065776  8578 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:17.067389  8578 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:17.068584  8578 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:17.073004  8578 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:17.075208  8578 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 13:04:17.077497  8578 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:17.060717  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.083747  8729 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:16.969444 15167 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.084813  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.077064 15237 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:17.087816  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.093847  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.096463 15068 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.049437 15292 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:16.969750 15278 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.119568  8657 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
W20260709 13:04:17.123541  8651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2933) took 1910 ms (client timeout 1530 ms). Trace:
W20260709 13:04:17.123992  8651 rpcz_store.cc:269] 0709 13:04:15.213313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.213502 (+   189us) service_pool.cc:224] Handling call
0709 13:04:17.123307 (+1909805us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54001d1fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:17.123467 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":521088256}
I20260709 13:04:17.125876  8579 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:17.127592  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.129611  8652 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:17.091375 15250 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.095750  8727 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:17.132055  8729 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:17.088056  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 228208us
W20260709 13:04:17.145954  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 277146us
I20260709 13:04:17.147063  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:17.148283 15241 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:17.149856  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:17.150663  8727 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:17.146021  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.151629  8729 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:17.156128  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:17.093720 15253 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 13:04:17.093818 15173 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:04:17.093524 15210 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 13:04:17.122618 15292 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.183691  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.184587  9328 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.188169  9329 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:04:17.172293 15207 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:17.207186  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:17.209585  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53555.1us
I20260709 13:04:17.202117  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.176054 15280 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.212244  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.213973  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:17.183359 15238 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.735s	user 0.008s	sys 0.011s
I20260709 13:04:17.230417  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:17.232616  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:17.234277  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.236979  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:17.178296 15279 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:17.209251 15276 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.221899  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.250926  9330 raft_consensus.cc:2410] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 18 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:04:17.239688  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.252283  9325 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:17.194257 15207 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:17.215688 15287 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:17.232661 15238 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.785s	user 0.008s	sys 0.011s
I20260709 13:04:17.268900  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.286526  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.276628  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.289374  9324 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:17.298890  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.299921  9330 raft_consensus.cc:2368] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16.
I20260709 13:04:17.234587 15136 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.240710 14765 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.302153 15306 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:17.261133 15279 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.292999 15287 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.370108  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:17.279388 15276 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.309342  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.310892  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.355643  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.317655  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.337268  9328 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:17.371026  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1108) took 1524 ms. Trace:
W20260709 13:04:17.317957 15293 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
I20260709 13:04:17.421244  9330 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:17.425520  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:17.309417 15296 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 13:04:17.378569  8650 rpcz_store.cc:276] 0709 13:04:15.846953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.855828 (+  8875us) service_pool.cc:224] Handling call
0709 13:04:17.369934 (+1514106us) spinlock_profiling.cc:243] Waited 5.21 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:04:17.370963 (+  1029us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11983744}
I20260709 13:04:17.242589 15280 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:17.328783 15182 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.620s	user 0.010s	sys 0.002s
W20260709 13:04:17.305106 15295 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:04:17.432509  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:17.436630  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.334826 15278 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.437590  9328 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:04:17.304527 15303 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.340466 15292 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.279896 15103 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.363256 15306 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.345842 15167 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.441009  8656 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:17.453260  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.372645 15250 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.450441  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.423359 15238 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:17.304270 15058 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.136s
I20260709 13:04:17.425052 14963 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.462401  9329 raft_consensus.cc:2368] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15.
I20260709 13:04:17.316726 15165 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.428051 15302 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.467263  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:17.371474 14765 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.467736  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:17.333076 15200 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.439849 15040 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:17.452525 15210 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:17.480079  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:17.472479  8732 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:17.469380  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.446473 15237 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 13:04:17.491394  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:17.487607  9324 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:17.488903  9325 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:17.466150 15238 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:17.454788 15306 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:17.440222 15182 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.469367 15250 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.470180 15311 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:17.502079  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:17.436482 15207 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.455065 15303 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.534222  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.535040  8725 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:17.482785 15040 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:17.510663  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:17.511133 15280 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.512028 15308 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:17.544716  8575 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:17.545010 15308 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.545434 15308 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:17.546926 15203 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:17.547395 15203 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:17.547694 15200 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.547886  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.503260 15058 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:17.548439 15203 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:17.550004 15304 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 13:04:17.550242 15276 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.550531  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3030) took 1899 ms. Trace:
I20260709 13:04:17.550973  8728 rpcz_store.cc:276] 0709 13:04:15.650738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.650874 (+   136us) service_pool.cc:224] Handling call
0709 13:04:15.822940 (+172066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:16.552621 (+729681us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.550224 (+997603us) spinlock_profiling.cc:243] Waited 729 ms on lock 0x7b540032df54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:17.550463 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1676128000}
I20260709 13:04:17.551784  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.552573  8728 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260709 13:04:17.552961  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.507643 15311 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.521503 15301 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.523396 15103 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.524065  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:17.525000  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.525676 15291 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:04:17.524431 15190 consensus_meta.cc:287] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.560s	user 0.015s	sys 0.001s
I20260709 13:04:17.526225 15250 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:17.526360 15297 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.220s	user 0.001s	sys 0.001s
I20260709 13:04:17.526865 15238 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.528416 15010 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:17.529870 15165 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.530478  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.531121 15287 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.526057  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:17.525030  9330 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:17.543033  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:17.543789  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:17.518534 15207 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:17.554562 15324 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:17.549074 15058 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.562577  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.557694 15279 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:17.564354  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1241) took 1383 ms (client timeout 1586 ms). Trace:
I20260709 13:04:17.653996  8730 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
W20260709 13:04:17.653942  8733 rpcz_store.cc:269] 0709 13:04:16.181171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.249859 (+ 68688us) service_pool.cc:224] Handling call
0709 13:04:17.564126 (+1314267us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b5400064854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:17.564281 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47081216}
I20260709 13:04:17.642630 15275 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.565908  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3319) took 1018 ms. Trace:
I20260709 13:04:17.566829  8731 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:04:17.568290  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:17.569425  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.656577  8584 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:17.656810  8581 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:17.571102  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:17.655809  8657 rpcz_store.cc:276] 0709 13:04:16.547669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.755983 (+208314us) service_pool.cc:224] Handling call
0709 13:04:16.756455 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.124533 (+368078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.565549 (+441016us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54000c8ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:17.565836 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":810323712}
W20260709 13:04:17.571383  8658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2938) took 2312 ms (client timeout 1846 ms). Trace:
I20260709 13:04:17.656491  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:17.479135 15302 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.573542 15040 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:17.559820  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2880) took 1375 ms. Trace:
I20260709 13:04:17.576886  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.661064  8583 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:04:17.579465 15182 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.559095  8727 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260709 13:04:17.663275  8727 pending_rounds.cc:85] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 11
I20260709 13:04:17.663663  8727 pending_rounds.cc:107] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 13:04:17.663965  8727 raft_consensus.cc:2889] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:17.664022  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.560531  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:17.558907  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1237) took 1447 ms. Trace:
I20260709 13:04:17.665087  8579 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:17.583081  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:17.660791  8726 rpcz_store.cc:276] 0709 13:04:16.184733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.276415 (+ 91682us) service_pool.cc:224] Handling call
0709 13:04:16.276745 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:16.302044 (+ 25299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.559575 (+1257531us) spinlock_profiling.cc:243] Waited 9.78 ms on lock 0x7b5400203ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:17.559761 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22494976}
I20260709 13:04:17.632169  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:17.636983  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 113006us
I20260709 13:04:17.640347  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:17.645272  8576 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
W20260709 13:04:17.666357  8655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1111) took 1818 ms (client timeout 1921 ms). Trace:
I20260709 13:04:17.649348 15314 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.649504 15136 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:17.649991  8728 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:04:17.639907 15068 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:17.653147  8580 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:04:17.658038  8658 rpcz_store.cc:269] 0709 13:04:15.259258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.259393 (+   135us) service_pool.cc:224] Handling call
0709 13:04:17.571059 (+2311666us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b540025ba54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:17.571215 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1267357056}
I20260709 13:04:17.641453 15301 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.661545  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.584251  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:17.665006  8724 rpcz_store.cc:276] 0709 13:04:16.111416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.121551 (+ 10135us) service_pool.cc:224] Handling call
0709 13:04:17.558507 (+1436956us) spinlock_profiling.cc:243] Waited 62.1 ms on lock 0x7b54004218d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:17.558824 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":142793088}
I20260709 13:04:17.588532  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:17.647487  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 103826us
I20260709 13:04:17.674031 15103 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.674539 15103 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:17.675463  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 91300.2us
I20260709 13:04:17.677423 14963 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:17.678200 14963 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:17.678141 15249 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:17.679129 15249 consensus_queue.cc:1243] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:17.679734 15249 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:17.680533 15170 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:17.651075 15250 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:17.652501 15238 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.676821 15103 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.683378  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:17.666803  8655 rpcz_store.cc:269] 0709 13:04:15.847718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.855726 (+  8008us) service_pool.cc:224] Handling call
0709 13:04:17.557337 (+1701611us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:04:17.666263 (+108926us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12378368}
I20260709 13:04:17.669009  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.645653 15010 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:17.688516  8655 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:04:17.677538 14267 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.657291 15324 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.679433 14963 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.680953 15170 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:17.653291 15203 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:17.684950  8577 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:17.666518  9296 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:17.669312  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 86357us
I20260709 13:04:17.671209 15136 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.677382  8696 leader_election.cc:400] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:04:17.689659 14267 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.692204  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.703006  8696 leader_election.cc:403] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:04:17.693231  8580 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:04:17.684119  8728 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:17.671715 15303 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.687587 14765 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.701203  8582 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:04:17.701488 15058 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.672326 14893 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:17.685626  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:17.698973  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:17.703341  8658 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:17.689369 15250 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.733880  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:17.735212  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:17.717260  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3310) took 2339 ms. Trace:
I20260709 13:04:17.737320  8725 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:17.660604 15275 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.740549  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.717053  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.709200  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.730321  8581 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.705637 15136 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:17.743636  8582 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 19: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:17.758036 15311 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.758378  8575 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:04:17.759166  8582 raft_consensus.cc:1275] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:17.759697  8575 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:17.745219 14267 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.762080 15278 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:17.737051 15231 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:17.740381 15010 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.762750 15231 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:17.763422 15010 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:17.763758 15231 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.766932 15250 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:17.744308  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.768277  8725 raft_consensus.cc:2410] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 13:04:17.768586  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.768989 15275 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.769327  8579 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
W20260709 13:04:17.766925 15333 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.133s	user 0.002s	sys 0.000s
I20260709 13:04:17.746981 15302 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.750084  8728 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:17.762521 15278 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:17.774057  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:17.742770  8583 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:17.778030 15278 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.775141  8655 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:17.780742  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:17.781953  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:17.782137 15318 log.cc:927] Time spent T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.236s	user 0.000s	sys 0.003s
I20260709 13:04:17.713877 15301 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:17.783164  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.742410 14765 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:17.736673  8652 rpcz_store.cc:276] 0709 13:04:15.377493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.378703 (+  1210us) service_pool.cc:224] Handling call
0709 13:04:17.123596 (+1744893us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.133463 (+  9867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.717020 (+583557us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54001d1fd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:17.717158 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3958555904}
W20260709 13:04:17.812179  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.736292  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:17.715682 15324 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:17.741037 15058 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:17.742584 14893 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:17.812937  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:17.869026  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 56189.3us
I20260709 13:04:17.871182  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:17.853240  8584 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
W20260709 13:04:17.855489  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 119323us
I20260709 13:04:17.856101  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.862280  8724 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:17.864709  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:17.865657  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:17.888230  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:17.888392  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:17.889529  8581 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:17.891186  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:17.896461  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.897423  8581 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:17.855139 15278 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:17.717684 15170 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.902091  8582 raft_consensus.cc:1240] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:04:17.897490 15041 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:17.879315 14980 raft_consensus.cc:2764] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 13:04:17.932030  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.932952  8584 raft_consensus.cc:2410] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 13:04:17.905699  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.849329 15250 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
W20260709 13:04:17.942771  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 135742us
I20260709 13:04:17.936803  8658 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:17.945355  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:17.944370  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 249462us
I20260709 13:04:17.946152  8732 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:17.946336  8581 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:17.938366  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:17.864508 15301 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:17.892917 15237 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:17.942065  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2672) took 1152 ms. Trace:
I20260709 13:04:17.951192  8578 rpcz_store.cc:276] 0709 13:04:16.789348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.078338 (+288990us) service_pool.cc:224] Handling call
0709 13:04:17.078615 (+   277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.078968 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.079000 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:17.085609 (+  6609us) log.cc:844] Serialized 26 byte log entry
0709 13:04:17.363708 (+278099us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:17.363770 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:17.363865 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:17.781568 (+417703us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:17.937022 (+155454us) raft_consensus.cc:1685] finished
0709 13:04:17.937047 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:17.941994 (+  4947us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7012,"spinlock_wait_cycles":63977856,"thread_start_us":206285,"threads_started":1,"wal-append.queue_time_us":447786}
I20260709 13:04:17.922964 14961 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:17.944823  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:17.947768  8582 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:17.950067  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:17.945348 15297 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 13:04:17.947870 15301 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:17.892273 15300 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 13:04:17.892587 14898 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
W20260709 13:04:17.957655  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:17.986974  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.953771 14961 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:17.965605 15240 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:17.958745 14980 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:17.967967 15068 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:17.970624 15040 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:17.959774 15170 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.949298 15103 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:04:18.000811 15314 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:18.002568 15222 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:18.022609 15222 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.026860  8577 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:04:18.028855 15041 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:18.029314 15041 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:18.030167 15041 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.030802 15308 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.031543 15041 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:18.034168  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:18.035408 15276 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:18.036203 15276 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.036578 15050 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:18.038141 15050 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:04:18.038831  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.038893 14748 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:18.039507 14748 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.004041 15306 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.989934 14893 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.040532 14748 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.003227 14980 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:18.042073 14748 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:18.042292 14980 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.043891  8657 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:18.013540  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.046552 14688 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:18.046924 15306 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:18.047704  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:18.048691  8577 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:17.986907 15301 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:18.037628 15276 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:17.992312 14961 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:18.050868  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:18.053902  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.068300 15200 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:18.068825 15200 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:18.069811  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.069281 15200 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:18.070873 14962 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:18.023649 15222 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.075368 15222 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:18.084113  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1105) took 2238 ms. Trace:
I20260709 13:04:18.086026 15211 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:18.086627 15211 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.087631 15211 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.087790 15301 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.015738  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.055629  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:18.101977  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46412.4us
I20260709 13:04:18.104259  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:18.084676  8656 rpcz_store.cc:276] 0709 13:04:15.846132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.846286 (+   154us) service_pool.cc:224] Handling call
0709 13:04:17.341227 (+1494941us) spinlock_profiling.cc:243] Waited 7.54 ms on lock 0x7b5800212d98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:04:17.341545 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.452503 (+110958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.083772 (+631269us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b5400352ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:18.084025 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":245658624}
I20260709 13:04:18.096890  8696 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.107049  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.109548  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:18.110049  8575 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:18.033215 15308 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:18.114928  8622 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:18.124495  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.107905 15324 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:18.127772  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:18.151760  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.159466  8729 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:18.112908 15103 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:18.112651 15211 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:18.162350  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47507.2us
I20260709 13:04:18.115036 14906 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:18.105751 15289 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 13:04:18.098464 14962 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:04:18.117379 15241 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:18.114995 15237 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:18.177670 15306 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:18.111122 15003 consensus_queue.cc:1059] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 13:04:18.181758 15230 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:04:18.185629  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.156208 15324 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.215523  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:18.154656 15240 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:18.199276 14906 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.202250 15103 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:18.203939 15241 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.220217 14906 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.206983 15306 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.217160 15324 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.193668 14961 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:18.216670 15207 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.565s	user 0.011s	sys 0.001s
I20260709 13:04:18.215385 14963 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:18.201911 15300 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
W20260709 13:04:18.089565 15212 consensus_queue.cc:1175] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1dc41db601fe4b91aa6d461b504da0e7" responder_term: 16 status { last_received { term: 15 index: 8 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:04:18.224198 15230 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.226733 15241 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:18.248760  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 160291us
I20260709 13:04:18.249636  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:18.236317 15306 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:18.252803  8697 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 2.309s (SENT) [suppressed 25 similar messages]
I20260709 13:04:18.243939 15203 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:18.242079 14963 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:18.248150 15230 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:18.270117 15301 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:18.273626 15040 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:18.273857 15262 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:18.260275 14505 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:18.257319 14770 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:18.275830 14963 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.276731 14770 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.279374 14770 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:18.279378 15136 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.281723 14961 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:18.283793 15136 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:18.277140 14963 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:18.290709  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1410) took 1080 ms. Trace:
I20260709 13:04:18.291262  9329 rpcz_store.cc:276] 0709 13:04:17.211502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.211922 (+   420us) service_pool.cc:224] Handling call
0709 13:04:18.290434 (+1078512us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54003b01d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:18.290631 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":542093184}
I20260709 13:04:18.284080  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:18.285547 15200 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.293802  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:18.301024  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:18.304503  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:18.305550  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:18.306115  8725 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:18.320220  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:18.331700  8579 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:18.307287 15103 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.340197  8726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:18.340505  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3033) took 1925 ms. Trace:
I20260709 13:04:18.340632  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.340943  8726 raft_consensus.cc:606] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.341370  8726 raft_consensus.cc:614] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:18.341327  8657 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:18.340945  8731 rpcz_store.cc:276] 0709 13:04:16.414770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.415986 (+  1216us) service_pool.cc:224] Handling call
0709 13:04:17.566374 (+1150388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.656323 (+ 89949us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.340270 (+683947us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400064850. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:18.340445 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2624798976}
I20260709 13:04:18.349550 15262 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:18.349854  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:18.350095 15130 raft_consensus.cc:3055] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:18.350478 15130 raft_consensus.cc:740] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.356361  8577 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:18.358511  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:18.356969 14770 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:18.398017  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.400887  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.401674  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:18.375244 14770 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:18.374986 15130 consensus_queue.cc:260] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.411890 14770 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:18.412724 15230 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.415441 15230 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:18.418006  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:18.420594 15345 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
W20260709 13:04:18.405357  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.443004  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.443987 14980 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:18.444422  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.406133  8654 raft_consensus.cc:2368] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:04:18.421582 15200 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:18.422765 14770 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.445905  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2957) took 1188 ms. Trace:
W20260709 13:04:18.446439  8548 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.676s (SENT)
I20260709 13:04:18.447441 14770 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:18.446329  8655 rpcz_store.cc:276] 0709 13:04:17.261992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.886636 (+624644us) service_pool.cc:224] Handling call
0709 13:04:17.887044 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.894263 (+  7219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.894312 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:17.901536 (+  7224us) log.cc:844] Serialized 26 byte log entry
0709 13:04:18.067961 (+166425us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b1800357fa0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:18.201639 (+133678us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:18.201706 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.201815 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:18.419922 (+218107us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:18.445674 (+ 25752us) raft_consensus.cc:1685] finished
0709 13:04:18.445695 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:18.445827 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4885,"raft.queue_time_us":2429,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":11691,"spinlock_wait_cycles":43747456,"thread_start_us":133413,"threads_started":1,"wal-append.queue_time_us":324213}
I20260709 13:04:18.423164  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:18.426391  8728 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:04:18.426944 15349 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 13:04:18.449440 15276 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:18.429682 14906 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.452075 14906 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:18.453063  8724 raft_consensus.cc:2368] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17.
I20260709 13:04:18.376744 15300 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:18.453737 15300 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.454152 15280 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:18.454620 15280 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:18.431903  8659 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:18.458972 15130 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:18.459992 15348 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.197s	user 0.001s	sys 0.001s
I20260709 13:04:18.459828 14748 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:18.456389 15280 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.463577 15280 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:18.433897  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.433744  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:18.466574  8656 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 13: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:18.429819  8731 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:18.454702 15300 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.467661  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1447) took 1429 ms. Trace:
I20260709 13:04:18.449777  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.468629  8650 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
W20260709 13:04:18.469894  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 829513us
W20260709 13:04:18.471426  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:18.473259  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.474133 15311 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:18.475211 15311 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:18.467221  8625 leader_election.cc:400] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17.
I20260709 13:04:18.471357  8583 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:18.468145  9322 rpcz_store.cc:276] 0709 13:04:17.038011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.038171 (+   160us) service_pool.cc:224] Handling call
0709 13:04:17.048959 (+ 10788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.216391 (+167432us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:17.459676 (+243285us) log.cc:844] Serialized 14 byte log entry
0709 13:04:17.766231 (+306555us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:18.466447 (+700216us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b180028b4c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:04:18.466708 (+   261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:18.466767 (+    59us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:18.466822 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.467421 (+   599us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54002148d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:18.467474 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:18.467605 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20670,"prepare.queue_time_us":156133,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":38,"raft.queue_time_us":828999,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":403307904,"thread_start_us":374049,"threads_started":2,"wal-append.queue_time_us":173405}
I20260709 13:04:18.508509  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:18.509507  8576 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
W20260709 13:04:18.510653  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43612.5us
W20260709 13:04:18.513087  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 846532us
W20260709 13:04:18.513535  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 69197.1us
I20260709 13:04:18.514384  8550 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.518743  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:18.521123  8575 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:18.522068  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:18.526702  8731 raft_consensus.cc:606] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.527161  8731 raft_consensus.cc:614] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:18.526916  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.530141  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:18.404428  8725 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:18.533321  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:18.534582  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:18.539297  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:18.545545  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:18.545851  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:18.546409  8578 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:18.546581  8659 raft_consensus.cc:2410] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 13:04:18.547991  9294 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:18.551075  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.538165  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.562548  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:18.563364  8577 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.565908  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.551105  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.578667  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:18.582285  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:18.604876  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.606525  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.607465  8659 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:18.608408  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:18.608729  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:18.609102  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:18.609494  8725 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:18.615079  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:18.547693 15308 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:18.551309 15237 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:18.617379  8578 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:18.570389 15212 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:18.575678 15182 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:18.578081 15275 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:18.527840 15279 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.093s
I20260709 13:04:18.534826 15104 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 13:04:18.539633 14906 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 13:04:18.608443 15071 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.632306 15308 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.654371  8725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:18.617923 15292 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:18.650635  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:18.648797 15342 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260709 13:04:18.666641  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 85146.8us
I20260709 13:04:18.667255  8725 raft_consensus.cc:606] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.676440  8725 raft_consensus.cc:614] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:18.666965  8577 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:18.677546  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.675510  8622 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:18.680687  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 100640us
I20260709 13:04:18.678737  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.609165 14963 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 13:04:18.653299 14688 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:18.658022 15275 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.660845 15314 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:18.661221 15212 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.641577 15190 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 13:04:18.674016 15308 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:18.696211  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.665092 15182 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:18.726267 15182 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.675163 15199 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.705937  8659 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:18.694980 15136 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:18.727854 15199 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.728765 15136 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:18.729866 15136 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.730811 15199 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:18.723711  8725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:18.671895 15071 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.732693  8725 raft_consensus.cc:606] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.733115  8725 raft_consensus.cc:614] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:18.663930 14947 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 13:04:18.735710  8731 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:18.736343 15071 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:18.736989  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:18.738026  8577 raft_consensus.cc:2410] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 17 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 13:04:18.738281 15241 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.694727 15275 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:18.739032  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:18.739727  8583 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:18.740664 15241 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:18.666003 15040 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 13:04:18.694664 15292 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:18.744158 15292 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.746191  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:18.746893  8579 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:18.752640  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.700330 15103 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:18.704531 15237 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:18.761279  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:18.761976  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65850.6us
I20260709 13:04:18.700141 15314 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.766888  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:18.772606  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:04:18.762950 15306 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.503s	user 0.005s	sys 0.005s
I20260709 13:04:18.774639 14770 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:18.789353  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.776494  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:18.830134  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 53730.4us
I20260709 13:04:18.764926 14963 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:18.799052  8579 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
W20260709 13:04:18.851353  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68580us
I20260709 13:04:18.852093  8582 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:18.781191 15104 consensus_queue.cc:1048] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:04:18.765053 15190 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:18.774449 15314 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:18.800542 15324 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.555s	user 0.014s	sys 0.000s
W20260709 13:04:18.861552  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 89081.6us
I20260709 13:04:18.781572 15222 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:18.852065 14963 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.795591 15103 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:18.831446 14688 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:18.795127 15003 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:18.859602 15130 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.863303 15190 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.895886  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:18.831073 15306 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.868909 15287 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:18.852187 14770 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:18.887686 14963 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:18.909934 15190 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:18.874339 15291 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 13:04:18.924259  8650 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:18.912858 15363 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.857297 15212 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.913628 15287 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:18.868331 15324 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:18.935968  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 384915us
I20260709 13:04:18.942013  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.951108 15362 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:18.951545 15359 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 13:04:18.952945 15357 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:04:18.953557  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.956178  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:18.957504 15362 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:18.961804 14688 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:18.962366 15364 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.962935 15364 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.965323 15364 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:18.967216  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:18.968345  8732 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:18.971195  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:18.972661 15364 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:18.973256 15364 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.974011 15364 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.934034 15104 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:18.979225 15104 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.980623 15361 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:18.995368  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:18.926976 15203 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:18.910506 15003 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:18.996028  8655 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:18.998440  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:19.003597  8653 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:18.998625  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:18.999827  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.913426 15130 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.007534  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:18.937584 15287 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:19.016973  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:19.018231  8653 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:18.999039 15365 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.995285 14893 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.932560 15199 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 13:04:18.988812 15362 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:18.933813 15363 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.040670  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.041579  8732 raft_consensus.cc:2410] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:04:19.042265 15363 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.044100 15104 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.045985 15324 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.046762 15306 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.046921  8655 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
W20260709 13:04:19.022212 15345 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:04:19.047883  8658 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:19.048589  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:19.048761  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 433712us
I20260709 13:04:19.049947  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:19.026732 15358 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 13:04:19.068071  8653 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:19.057529  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:19.059139  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.078505  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:19.091571  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.096338  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.098623  9326 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:19.102246  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:19.104218  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.106365  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.105942  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.111084  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.114796  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.116147  9327 raft_consensus.cc:2410] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 16 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:04:19.063459 15362 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:19.121105  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.122275  9323 raft_consensus.cc:2410] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 18 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:04:19.063560 14827 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 13:04:19.127969 15362 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.131338 15362 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.049288 15365 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.155881  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.023737 15103 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.189931  9323 raft_consensus.cc:2376] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 16. Re-sending same reply.
I20260709 13:04:19.165417 15275 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.165828 15374 consensus_queue.cc:1048] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:19.169847 15314 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.164053 14963 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.173548 15364 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.056631 14893 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.177595 15287 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.180840 15324 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:19.186276 14770 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.204269  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.205775  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.206552  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.209667  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.211248  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.136828 15190 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.214493  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:19.219698  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 314506us
I20260709 13:04:19.244204  9330 raft_consensus.cc:2368] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21.
I20260709 13:04:19.252769  9322 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:04:19.274000  9326 raft_consensus.cc:2368] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 17.
W20260709 13:04:19.212395 15301 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.010s	sys 0.010s
I20260709 13:04:19.261263  9324 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:19.261816  9328 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:19.274973  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.275996  9326 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:19.277204  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.279009  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.282692 15324 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:19.283763  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.285352  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:19.285468  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.287324  9331 raft_consensus.cc:2410] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 17 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 13:04:19.287233  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.289680 15324 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:19.289898 15199 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:19.292018  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.291149 15199 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.293444  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:19.281870  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.296913  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:19.300117  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 241014us
W20260709 13:04:19.280745 15301 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.000s	user 0.010s	sys 0.010s
I20260709 13:04:19.314345  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.293345 15199 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:19.298031  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.314198  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.306349  9324 raft_consensus.cc:1240] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:19.314627  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.315140  9323 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:19.306097  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.318879  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:19.296101 15308 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.001s
I20260709 13:04:19.291234 15324 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.325656  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.316555  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.325520  9326 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:04:19.325613  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:19.323611  9328 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:19.326269  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.329793  9323 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:19.331498  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:19.332337  8655 raft_consensus.cc:2368] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:04:19.337574  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:19.335614  8655 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 13:04:19.339608  8655 pending_rounds.cc:85] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 8
I20260709 13:04:19.339975  8655 pending_rounds.cc:107] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 13:04:19.340405  8655 raft_consensus.cc:2889] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:19.315378 15376 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:19.323292 15361 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:04:19.333130  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.369678  9328 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:19.371495  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:19.338136 14770 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.377835  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:19.389667 15367 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 13:04:19.390034 15314 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.394608 15364 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.395047 15190 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.346709 15292 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:19.395936 14963 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.396358 15292 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.397261 15365 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.398826 15292 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.324214 15308 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.366042 15324 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:19.365988 15376 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:19.419260  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.419872  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.422631  8725 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:19.424554  8697 leader_election.cc:400] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 17.
I20260709 13:04:19.425755  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.444046  9331 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:19.470234  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.438316  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:19.425817  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.491497  8726 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:19.407280 14893 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:19.458799 15389 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:19.514768  9325 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:19.420948 15386 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.458014 15384 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 13:04:19.423976 15376 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:19.471998 15275 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.524542  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.484871 15364 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.563470  9325 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:04:19.446185 15393 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:19.563925  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 240405us
I20260709 13:04:19.555444  8725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:19.485929 15287 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:19.564404  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 138754us
I20260709 13:04:19.564777  8725 raft_consensus.cc:606] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:04:19.565303  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.423295 15324 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:19.567214  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 248393us
I20260709 13:04:19.568594  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:19.568735  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.569613  9328 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:19.570302  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:19.571753  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.577605  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:04:19.579802  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:19.580724  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.582527  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.583158  9328 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:19.575393  9325 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:19.586149  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.588586  8726 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:19.570983 15361 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:19.591140  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:19.592900 14948 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.593902  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.575932 15363 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.568358 15364 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.578151 15385 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:19.602062  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:19.602807 15403 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.603250  8651 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:19.613014 15393 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:19.617700  9328 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:19.619581  8698 leader_election.cc:400] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14.
I20260709 13:04:19.613279  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.625743 15364 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.631650  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:19.562129 15386 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:19.634792  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 54161.8us
I20260709 13:04:19.613719 15385 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:19.640365 15388 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
I20260709 13:04:19.639706  8697 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.623091 15308 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.633325 15400 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:19.648032 15400 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:19.649051 15400 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.650998  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:19.658294 15393 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:19.658960 15393 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.661300 15393 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:19.662858 15377 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
W20260709 13:04:19.666247 15370 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.539s	user 0.003s	sys 0.000s
I20260709 13:04:19.669828  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3331) took 1265 ms. Trace:
I20260709 13:04:19.662161 15199 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:19.675465  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 75373.9us
I20260709 13:04:19.685637  9327 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:19.615023 14948 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.633242 15404 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:19.689328  8726 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 13:04:19.689606  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 49977.9us
I20260709 13:04:19.690351  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:19.641593 15385 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:19.670220  8657 rpcz_store.cc:276] 0709 13:04:18.404328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.404872 (+   544us) service_pool.cc:224] Handling call
0709 13:04:18.411590 (+  6718us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:18.412008 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:18.634345 (+222337us) log.cc:844] Serialized 14 byte log entry
0709 13:04:18.922356 (+288011us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:18.922849 (+   493us) log.cc:844] Serialized 26 byte log entry
0709 13:04:18.922990 (+   141us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:18.923033 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.923097 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:19.665907 (+742810us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b2400220ec8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b6809d 00007f140979a10e 00007f14097b188c 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221
0709 13:04:19.669479 (+  3572us) raft_consensus.cc:1685] finished
0709 13:04:19.669619 (+   140us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:19.669763 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2644,"prepare.queue_time_us":242644,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":46,"raft.queue_time_us":2917,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":8166,"spinlock_wait_cycles":137540992,"thread_start_us":468813,"threads_started":2,"wal-append.queue_time_us":451983}
I20260709 13:04:19.698442  8730 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:19.703564  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:19.703975  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.705891  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.707597  8579 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:19.707723  8697 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.708781  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:19.709005  8655 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:19.709717  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.710222  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:19.710901  8579 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:19.712126  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:19.712612  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.713053  8576 raft_consensus.cc:2393] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:19.716107 15308 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:19.713496 15403 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:19.702124 15395 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.718187  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.719200  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.723104  9294 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:19.716699 15406 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.713414 15280 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:04:19.737624 15392 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.156s	user 0.002s	sys 0.000s
I20260709 13:04:19.739208 15386 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.724472 15393 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:19.741561 15308 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:19.742127 15308 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:19.742564 15308 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:04:19.724640 15343 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.460s	user 0.006s	sys 0.000s
I20260709 13:04:19.713701 15275 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:19.707136 15363 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:19.737123 15280 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:19.745935 15363 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.748368 15165 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:19.748478 15363 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.730824  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:19.748785  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.728638  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.728796 15372 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.007s
I20260709 13:04:19.735662 15401 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.730367 15404 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:19.753937  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.746467 15280 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:19.749050 15165 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:19.756294  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:19.760775  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:19.751168  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.726447 14948 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.758205  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.737869 15324 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.762045  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:19.765880 15403 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:19.774189  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.779296  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.768245 15363 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:19.780624 15363 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.780419 15403 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.780865 15397 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:19.729367 15364 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 13:04:19.756780 15393 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:19.781126 15363 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:19.789378 15407 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:19.761710 15406 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:19.766862 15210 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.792773  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.793480  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.793789  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.793689  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.797175  9322 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:19.783529 15372 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:19.795564  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.795907  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.800721  9330 raft_consensus.cc:2368] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21.
I20260709 13:04:19.806947  9322 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:19.780619 15399 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:19.809103  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:19.777251 14948 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:19.810055  9329 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:19.755929 15401 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.817075 15401 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.783135 15275 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:04:19.822685 15403 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.810035 15393 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:19.823647 15199 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.810258 15103 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.828320 15385 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.834996 15406 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:19.842165 15406 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.809219 15210 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.838254 15393 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.840425 15103 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.849148  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 2969) took 1127 ms. Trace:
I20260709 13:04:19.808885 15382 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.832770 15376 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.856844  8650 rpcz_store.cc:276] 0709 13:04:18.721678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.722494 (+   816us) service_pool.cc:224] Handling call
0709 13:04:18.722847 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.929991 (+207144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.848878 (+918887us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540013a954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:19.849085 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462668672}
I20260709 13:04:19.834959 15400 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:19.781001 15165 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:19.872624 15396 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.416s	user 0.001s	sys 0.000s
I20260709 13:04:19.873709  8732 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:19.874897 15376 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.876137 15385 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:19.876832  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.878746  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:19.879908  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.880944  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.880978 15400 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.896467 15400 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.883018 15103 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.877316  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:19.873724 14733 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:19.898154 14733 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:19.885691  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:19.899243  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:19.887504  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.901115  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:19.902047  9330 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:19.892302  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.889292 14827 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:19.904271  8733 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:19.905660  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:19.906977  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 111112us
I20260709 13:04:19.903745  8729 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:19.911278  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:19.916019  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:19.905714 14733 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.918733 15382 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:19.919327 14733 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:19.914995  8548 leader_election.cc:400] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21.
I20260709 13:04:19.926546  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:19.927224  8730 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:19.928179  8733 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:19.904099 14827 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:19.909548 15275 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:19.932577  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.917333 15407 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:19.914444 14688 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:19.920946 15207 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:19.928730 15165 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:19.939196  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:19.945679 14948 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 13:04:19.936527 15414 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:19.951361 15210 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:19.946942 15400 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:19.956113 15400 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.956390  9330 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:19.959466  8730 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:19.959684  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:19.958247 15400 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:19.962261 15407 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.953298 14827 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.984666 14827 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:19.955334 15414 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:19.991766 15414 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:19.965106 15173 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:19.993345 15173 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:19.994378 15173 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:19.996690 15211 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:19.997159 15211 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:19.997534 15415 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:19.998039 15211 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.010638  9330 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:04:20.015826  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:19.966539 15308 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.010061 15211 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:20.010262 15412 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:19.970983 15362 consensus_queue.cc:1059] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 13:04:20.040364  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:20.041898  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:19.998068 15415 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.050117  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:20.051841  8576 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:19.974356 15363 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.058805  9323 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:20.075469  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2925) took 1616 ms. Trace:
I20260709 13:04:20.072528 15182 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:20.077208 15210 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:20.103566 15407 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.104141 15407 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.106487 15407 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.052992 15412 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.086103 15414 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.134037  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.101325 15362 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.148018 15415 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.150970  8582 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:20.152113  8584 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:20.102602 15182 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.157119  9329 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:20.163944  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2705) took 1419 ms. Trace:
I20260709 13:04:20.167130  8582 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:20.143473 15130 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:20.171662 15422 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:04:20.164470  8583 rpcz_store.cc:276] 0709 13:04:18.744866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.749154 (+  4288us) service_pool.cc:224] Handling call
0709 13:04:18.766953 (+ 17799us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.767165 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:20.055481 (+1288316us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.163626 (+108145us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.163674 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:20.163718 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.163782 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.163886 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1268374,"prepare.queue_time_us":1289730,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":8380,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":1253,"thread_start_us":107675,"threads_started":2,"wal-append.queue_time_us":106345}
I20260709 13:04:20.164454 15393 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 13:04:20.178864 15412 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.181679  9331 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:20.136096 15414 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.192130 15412 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.192920  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:20.192739 15412 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:20.193588 15425 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.057s	user 0.000s	sys 0.002s
I20260709 13:04:20.193701  8583 raft_consensus.cc:2368] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14.
I20260709 13:04:20.195304 15182 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.197535 15412 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.197247 15280 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.143827 15362 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.202023  9294 leader_election.cc:400] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14.
I20260709 13:04:20.256260  9294 leader_election.cc:403] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
W20260709 13:04:20.259550  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 59250.6us
I20260709 13:04:20.165639 15103 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 13:04:20.283149  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2924) took 1824 ms. Trace:
I20260709 13:04:20.283586  8724 rpcz_store.cc:276] 0709 13:04:18.458580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.492610 (+ 34030us) service_pool.cc:224] Handling call
0709 13:04:20.282401 (+1789791us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:04:20.282616 (+   215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.282793 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:20.282817 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.282864 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:20.282894 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.282948 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.283088 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61069568}
I20260709 13:04:20.280509 15412 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.293133 15412 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.294435 15414 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.266898 15365 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:20.295466 15412 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.280948 15407 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.296789 15407 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.285241  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2703) took 1736 ms. Trace:
I20260709 13:04:20.298941  8580 rpcz_store.cc:276] 0709 13:04:18.548883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.549252 (+   369us) service_pool.cc:224] Handling call
0709 13:04:18.549569 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.549780 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:18.585134 (+ 35354us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.037712 (+1452578us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.266125 (+228413us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.266193 (+    68us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:20.266243 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.285036 (+ 18793us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.285181 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1322712,"prepare.queue_time_us":2793988,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":2034,"raft.queue_time_us":85697,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":15918,"spinlock_wait_cycles":9422080,"thread_start_us":395477,"threads_started":3,"wal-append.queue_time_us":71324}
I20260709 13:04:20.258780 15210 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.306015 15210 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.262830 15397 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:20.263352 15308 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.303854 15182 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.255435 15400 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.312251 15414 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.308264 15363 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.317919  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1276) took 1183 ms. Trace:
I20260709 13:04:20.318873 15400 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.322114  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2697) took 2025 ms. Trace:
I20260709 13:04:20.322517  8581 rpcz_store.cc:276] 0709 13:04:18.297168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.297282 (+   114us) service_pool.cc:224] Handling call
0709 13:04:18.297843 (+   561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.298039 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:18.543074 (+245035us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.001255 (+1458181us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b18003a7440. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:04:20.001472 (+   217us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.309299 (+307827us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.321694 (+ 12395us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.321743 (+    49us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:20.321787 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.321868 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.322055 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":105111,"prepare.queue_time_us":1560487,"prepare.run_cpu_time_us":149,"prepare.run_wall_time_us":2460,"raft.queue_time_us":29656,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":18717,"spinlock_wait_cycles":81178240,"thread_start_us":542074,"threads_started":2,"wal-append.queue_time_us":304612}
I20260709 13:04:20.330605  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:04:20.330763 15391 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.875s	user 0.006s	sys 0.008s
I20260709 13:04:20.318326  8577 rpcz_store.cc:276] 0709 13:04:19.134597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.140163 (+  5566us) service_pool.cc:224] Handling call
0709 13:04:19.161804 (+ 21641us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.162077 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:20.167643 (+1005566us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.308105 (+140462us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.317579 (+  9474us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.317629 (+    50us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:20.317683 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.317742 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.317857 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":904498,"prepare.queue_time_us":916443,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":3527,"raft.queue_time_us":11695,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":4090,"thread_start_us":250134,"threads_started":2,"wal-append.queue_time_us":144591}
I20260709 13:04:20.335271  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3055) took 1584 ms. Trace:
I20260709 13:04:20.336467  8576 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:20.337204 15421 raft_consensus.cc:3055] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:20.337790 15421 raft_consensus.cc:740] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:20.075953  8728 rpcz_store.cc:276] 0709 13:04:18.458991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.484093 (+ 25102us) service_pool.cc:224] Handling call
0709 13:04:19.999238 (+1515145us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b5800180098. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:04:19.999441 (+   203us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.999671 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:20.000002 (+   331us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.055305 (+ 55303us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.058102 (+  2797us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.072468 (+ 14366us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.072526 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:20.072574 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.072641 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.075383 (+  2742us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32176,"prepare.queue_time_us":21968,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":29869,"raft.run_wall_time_us":3027,"spinlock_wait_cycles":79940608,"thread_start_us":36590,"threads_started":1}
I20260709 13:04:20.339845  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:20.344038 15401 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:20.335767  8727 rpcz_store.cc:276] 0709 13:04:18.750927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.751040 (+   113us) service_pool.cc:224] Handling call
0709 13:04:18.765324 (+ 14284us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:18.795819 (+ 30495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:18.795874 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:18.796530 (+   656us) log.cc:844] Serialized 26 byte log entry
0709 13:04:19.139352 (+342822us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b18002d6a00. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:19.318431 (+179079us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:19.318510 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.318625 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:20.330045 (+1011420us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:20.334982 (+  4937us) raft_consensus.cc:1685] finished
0709 13:04:20.335011 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.335197 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21373,"raft.queue_time_us":53122,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":18552,"spinlock_wait_cycles":9519360,"thread_start_us":178715,"threads_started":1,"wal-append.queue_time_us":600381}
I20260709 13:04:20.340893 15203 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:20.341156 14980 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:20.349311 15203 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:20.351347 15407 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.354235 15362 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:20.339219 15182 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.356295 15182 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.348013 15401 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:20.348232 15280 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:20.342723 15414 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.361852 15210 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.357241 15401 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.368460 15376 raft_consensus.cc:993] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower a0d703ee18464475a290425976246555 to start an election
I20260709 13:04:20.369167 15376 raft_consensus.cc:1081] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Signalling peer a0d703ee18464475a290425976246555 to start an election
I20260709 13:04:20.370919 15414 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.342206 15421 consensus_queue.cc:260] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.372357 15421 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:20.372678 15378 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.007s	user 0.001s	sys 0.003s
I20260709 13:04:20.381748  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2700) took 1836 ms. Trace:
I20260709 13:04:20.382254  8575 rpcz_store.cc:276] 0709 13:04:18.545029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.548452 (+  3423us) service_pool.cc:224] Handling call
0709 13:04:18.548805 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.652573 (+103768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:18.652619 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:18.652675 (+    56us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:18.652726 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.381516 (+1728790us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54004407d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:20.381567 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.381689 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238066944}
I20260709 13:04:20.396852  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:20.398068  9328 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:04:20.398294  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.399111  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:20.399557  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:20.399883  9322 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:20.400267  8579 raft_consensus.cc:2393] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:20.397617  9329 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:20.398909  8584 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:20.417943 15182 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.418551 15182 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.420864 15182 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.422741  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:20.424142 15421 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:20.424574 15421 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.424932 15421 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:20.431546  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:20.430747 15383 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.038s	user 0.002s	sys 0.007s
I20260709 13:04:20.392838 15203 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.443589 14906 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:20.444089 14906 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:20.445000 14906 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.446300 14906 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:20.454450 15207 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.455183 15207 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:20.463639  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.464880 15182 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:20.466872 15306 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.467433 15306 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.468462 14980 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.469175 14980 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:20.469633 15136 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.470175 15136 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.472216 15136 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.472807 15136 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.441675 15397 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14.
I20260709 13:04:20.474118 15182 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:20.475286  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:20.475790 15381 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:20.476362 15381 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.463723  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1125) took 1430 ms. Trace:
I20260709 13:04:20.478466  8654 rpcz_store.cc:276] 0709 13:04:19.033850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.034012 (+   162us) service_pool.cc:224] Handling call
0709 13:04:19.034354 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.034589 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:19.059796 (+ 25207us) log.cc:844] Serialized 14 byte log entry
0709 13:04:19.283094 (+223298us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.391735 (+1108641us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:20.462998 (+ 71263us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b18004c8880. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:04:20.463220 (+   222us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.463301 (+    81us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:20.463388 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.463498 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.463645 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1177084,"prepare.queue_time_us":36613,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":1195047,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":4767,"spinlock_wait_cycles":34287616,"thread_start_us":221067,"threads_started":1,"wal-append.queue_time_us":330954}
I20260709 13:04:20.481591 15381 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.483757  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:20.485559 14505 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.484541  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:20.486115 14505 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.486501 15212 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:20.490217  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:20.491848  8653 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:04:20.486513  9330 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:20.498580  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.485231 15381 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:20.501348  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:20.502135  8653 raft_consensus.cc:3060] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:20.487035 15212 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:20.503656  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.504302  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.506906  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:20.510689  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.511463  8579 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:20.511662  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.512523  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.512751  8576 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:20.513617  8575 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:04:20.496166 15365 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:20.514742  8580 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260709 13:04:20.516202  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 801032us
I20260709 13:04:20.495823 14893 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:20.497524 15306 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.528604  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.529006  8579 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:20.473246 15136 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.535115  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:20.537388  8653 raft_consensus.cc:2468] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:20.538054  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.541371  8724 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 13:04:20.541905  8724 pending_rounds.cc:85] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 10
I20260709 13:04:20.542191  8724 pending_rounds.cc:107] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 13:04:20.542476  8724 raft_consensus.cc:2889] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:20.542590  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.543565  8653 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:20.547369  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.548251  8728 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:20.544981  8581 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:20.545943  8583 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:20.496038 14906 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 13:04:20.552671  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.495849 15414 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 13:04:20.540902 15407 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 13:04:20.567749 15407 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:20.568168 15407 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:20.547561  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.555410  8650 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:20.570266 15203 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.559113 15276 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:20.556962  8622 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:20.569103 15407 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.571094 15276 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.557042  8623 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:20.560108 15050 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.572763 15050 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.527495 15365 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.574870 15050 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.560611 15211 consensus_queue.cc:1059] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:20.561345  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.540304 15399 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 13:04:20.540773 15314 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 13:04:20.575178 15365 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:20.579247 14688 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:20.579808 14688 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:20.559664 15130 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.581770 15407 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:20.581902 15136 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.582371 15130 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.537653 14893 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.585912 14893 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.585959  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2929) took 1894 ms. Trace:
I20260709 13:04:20.587108  8731 rpcz_store.cc:276] 0709 13:04:18.692100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.692266 (+   166us) service_pool.cc:224] Handling call
0709 13:04:18.692610 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.736124 (+ 43514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.585650 (+1849526us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b5400064d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:20.585882 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98175232}
I20260709 13:04:20.588747  8651 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:04:20.545181 15421 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.590982 14961 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:20.591459 15421 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:20.591727  9329 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:20.592671 15303 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:20.502856 15381 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.571100  8650 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:20.595643 15381 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.555469 15212 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.596766 15212 raft_consensus.cc:3055] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:20.597200 15212 raft_consensus.cc:740] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:20.597925  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1422) took 1473 ms. Trace:
I20260709 13:04:20.598188 15212 consensus_queue.cc:260] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.600165 15212 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:20.600401 15280 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:20.602828  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45870.7us
I20260709 13:04:20.603636  8623 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:20.586560 14893 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:20.621345  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:20.595887 15427 log.cc:927] Time spent T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.486s	user 0.007s	sys 0.000s
I20260709 13:04:20.611861  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:20.623245  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:20.585069 15130 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.606004  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.628789  9330 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:20.635255  8724 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:20.629715  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.598331  9327 rpcz_store.cc:276] 0709 13:04:19.124432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.143517 (+ 19085us) service_pool.cc:224] Handling call
0709 13:04:19.685444 (+541927us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.686133 (+   689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.597730 (+911597us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b54006c3750. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:20.597851 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1218713344}
I20260709 13:04:20.601056 15280 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.569026 15207 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.642450  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:20.647356  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.642984  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.611189  8655 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:20.641141  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.685523  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:20.685642  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.686458  8728 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:20.687382  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.629544 15136 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:20.690725 15136 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:20.691154 15136 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:20.691403  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.688087 15400 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.692752  8473 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 8 to 11, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 11 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:20.694100 15400 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.629315 15380 consensus_queue.cc:1059] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 13:04:20.696842  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:20.697449  8729 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:20.689857  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:20.699290  8724 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:20.688732 15280 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.703039  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:20.647441 14893 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:20.689158 15050 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:20.689770 15203 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:20.689158 15207 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.665330 15412 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.706512 15173 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.694784  8473 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 9 to 10, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 10 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260709 13:04:20.708384  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 118258us
I20260709 13:04:20.710583  8698 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:20.730903  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:20.711388  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.704489  8725 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:20.640779 15130 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:20.672431 14505 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.702795 15436 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:20.722263  8652 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:20.708500 15020 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:20.707319  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.732245  8581 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:20.724956 15173 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:20.737547 15384 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:20.738189 15394 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:20.740489  8654 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:20.791782  8654 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:20.740751 14963 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:20.741922  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.742323  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:20.742183 14906 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:20.744141 15136 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.746213 14893 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:20.753881 15387 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 1.169s	user 0.006s	sys 0.004s
I20260709 13:04:20.755348  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:20.757632 15364 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.764485  9325 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:04:20.819875  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:20.820501  9325 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:20.765158  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:20.767256  9323 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:04:20.766510  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:20.822685  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56275.1us
I20260709 13:04:20.823817  8550 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:20.776866  8728 raft_consensus.cc:2393] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:20.781725  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.794287  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:20.794932  9329 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:20.795552  9328 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:20.795449  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:20.797008  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 3060) took 1378 ms. Trace:
I20260709 13:04:20.827287  8732 rpcz_store.cc:276] 0709 13:04:19.418871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.419003 (+   132us) service_pool.cc:224] Handling call
0709 13:04:20.796861 (+1377858us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400065754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:20.796961 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1038345088}
I20260709 13:04:20.828583  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:20.829744  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:20.798606  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55657.6us
I20260709 13:04:20.804517  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:20.831483  8728 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:20.832115  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.809995  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.833323  8651 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:20.770982 15207 raft_consensus.cc:2804] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:20.837831 15207 raft_consensus.cc:697] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:20.838120 15003 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:20.811161  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 68935us
I20260709 13:04:20.838773 15207 consensus_queue.cc:237] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.840682 15415 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:20.841137 15415 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.841498 15415 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:20.845580  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50551.9us
I20260709 13:04:20.846613 15381 consensus_queue.cc:1059] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:20.847709 15381 raft_consensus.cc:3055] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:20.848068 15381 raft_consensus.cc:740] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:20.848850 15381 consensus_queue.cc:260] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.850065 15381 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:20.853072  8656 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:20.854513  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.855252  8656 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:20.856007  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.857026  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.857510 15364 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.860000 15364 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:20.830260  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.809209  8724 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:20.815315  8727 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:20.776331 15400 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:20.774398 15020 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:20.817325  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62116.7us
I20260709 13:04:20.791277 15384 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.805888  8729 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
W20260709 13:04:20.834252  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.881052  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:20.881956  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:20.816345 14827 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:20.846755  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:20.811724  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.889271  8657 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:20.867609  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.867835  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.868407  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.869446  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.892165  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.892467  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.892840  8579 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:20.861716  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:20.873230  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:20.874188  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.830324  8732 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:20.883566  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.884037  8581 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:20.894174  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.726248 15412 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.862126 15407 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:20.833041 15324 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:20.887571  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.791453 15436 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:20.905682  8724 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:20.907037  8724 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
W20260709 13:04:20.888381  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.896031  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:20.887804  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:20.908066  8732 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:20.888190  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:20.908707  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:20.877333 15020 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:20.898972  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.899313  8651 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:20.899607  8581 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:04:20.901463  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:20.909766  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:20.914422  8657 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:20.898476  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.915405  8650 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:20.884516 15384 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:20.897934 15361 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:20.923360  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.900189 14505 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:20.876782 15364 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 13:04:20.931017  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.934419  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:20.908993 15050 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:20.905565 15407 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:20.903837 14827 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:20.929514 14505 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.939105  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.939515  8651 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:20.944072  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:20.911600 15182 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:20.909848 15324 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:20.893739 14963 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:04:20.988351 15407 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:20.926870  8581 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7"
dest_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:41980
I20260709 13:04:20.989765  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:20.904302 15363 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 13:04:20.992954  8651 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:20.902385 15380 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 13:04:20.923032 15361 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.903833 15130 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 13:04:20.995404 14505 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:20.998528  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54576.1us
I20260709 13:04:20.985503 15394 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:20.944638 15137 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:20.944689 15050 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.007054  8581 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:04:21.027455  8581 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:20.921140 15376 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 13:04:21.008563  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.917644 15448 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.089s
I20260709 13:04:20.935290 15071 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 13:04:21.020251  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.020344  8657 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:21.011857 15182 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:21.035189  8656 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 13:04:20.995731  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 94290us
W20260709 13:04:21.004312 14980 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.531s	user 0.009s	sys 0.003s
I20260709 13:04:21.035938 14980 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.036013 15182 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:21.023943  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 115316us
I20260709 13:04:21.038142  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:21.008585 15401 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:21.009071  8651 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.042980  8651 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:20.996946 15364 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:21.028928 15050 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.029635 14827 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.039220  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.052907  8657 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 19. Current term is 20. Ops: [18.14-19.15]
I20260709 13:04:21.001497 15324 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:21.059577 15050 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.062248  8652 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:21.012089 15361 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:21.046226  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:21.073558  8732 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:21.073925 15136 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.074540 15136 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:21.074972  8732 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:04:21.075119 15136 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:21.076053  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.077747  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.078635  8651 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 13:04:21.079144  8651 pending_rounds.cc:85] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 10
I20260709 13:04:21.079012  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.079437  8651 pending_rounds.cc:107] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 13:04:21.079737  8651 raft_consensus.cc:2889] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:21.081557  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.083190  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:20.998584 15365 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:21.083209  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.026029 15137 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:21.088222  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.090267  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.042518 15401 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.095347  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.102905  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:21.104416  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.108801  8584 raft_consensus.cc:1240] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:21.099370  8581 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.129206  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.130465  8583 raft_consensus.cc:2393] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:21.084517 15212 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.134882  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.141690  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.140115  8582 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:21.073180 15361 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.094362 15137 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.090876 15173 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:21.064839 15380 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 13:04:21.101799 15401 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.166679  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.175714  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:21.168166  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.168154  8729 raft_consensus.cc:1240] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:04:21.178056  8583 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.084517 15324 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.192118  8575 raft_consensus.cc:2368] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20.
I20260709 13:04:21.195442  8581 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.196777  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:21.197695  9327 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:04:21.202152  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 181965us
I20260709 13:04:21.159950 15406 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:21.207422  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:21.197310 15451 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 13:04:21.211521  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3252) took 2500 ms. Trace:
I20260709 13:04:21.211341  8582 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:21.216629  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 178547us
I20260709 13:04:21.217532  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:21.217507  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:21.161423 15212 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.216220  8578 rpcz_store.cc:276] 0709 13:04:18.708114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.708300 (+   186us) service_pool.cc:224] Handling call
0709 13:04:18.708743 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.709037 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:20.036772 (+1327735us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.597292 (+560520us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b18003338e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f14097be4bc 00007f1406b0c48b 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a
0709 13:04:20.942062 (+344770us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.208103 (+266041us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:21.208170 (+    67us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:21.208234 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.208321 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.208453 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1655169,"prepare.queue_time_us":1691171,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":1871,"raft.queue_time_us":228087,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":30413,"spinlock_wait_cycles":10945408,"thread_start_us":304661,"threads_started":2,"wal-append.queue_time_us":64183}
I20260709 13:04:21.226019  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:21.210477 14770 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:21.211809 15050 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:21.229734  8583 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:21.227198  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.209043 15457 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.241734  9327 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:21.217980 15173 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:21.264559 14505 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.268477  8650 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:04:21.271461  8651 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:21.223897 15406 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:04:21.283429  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.223066 15363 consensus_queue.cc:1243] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:21.231590 15364 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:21.303970 14505 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.220595 15003 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 13:04:21.307010  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.221698 14963 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 13:04:21.255484 15453 raft_consensus.cc:2764] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:04:21.301126 15380 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:04:21.318423  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 93433.2us
W20260709 13:04:21.309334 15446 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20260709 13:04:21.336849  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:21.338006  9327 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:21.321892 15432 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:21.274806 15457 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.355636  9327 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:04:21.337312 15365 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.341743 15432 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.428169 15432 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.368040 15457 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.368732 15324 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.440171 14963 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:21.446933 14963 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.449087 14963 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:21.388196 15020 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.359807 15364 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.390259 15182 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:21.395629 15407 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.401190 15401 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.405953 15173 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.406563 14980 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:21.414239  8655 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:21.473008  8655 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:21.418373 15412 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.418967 15287 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:04:21.424404 15415 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.582s	user 0.006s	sys 0.003s
I20260709 13:04:21.424892  8584 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:21.427345 15212 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.334391 15467 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.100s
I20260709 13:04:21.439270  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:21.442862  8696 leader_election.cc:400] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20.
I20260709 13:04:21.484671  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.386444 15200 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:04:21.451624  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46483.3us
I20260709 13:04:21.354357 15361 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:21.477799 15384 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.536s	user 0.007s	sys 0.003s
I20260709 13:04:21.478003  8658 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:04:21.421131 15301 raft_consensus.cc:3055] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:21.450186 15376 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:21.487277  8584 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
W20260709 13:04:21.474516  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 248554us
W20260709 13:04:21.491067  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51890.7us
I20260709 13:04:21.464767 15324 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 13:04:21.482375 15415 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.481065  8655 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.488327 15413 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:04:21.491413 15381 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.620s	user 0.008s	sys 0.004s
I20260709 13:04:21.496443  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.488799 15200 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:21.477519 15173 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.494112 15406 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:21.497874  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.509150  8655 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.493273 15301 raft_consensus.cc:740] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:21.518291 15301 consensus_queue.cc:260] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.510142  8584 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:21.510645 14505 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:21.512070  8653 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:21.528033  8655 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:21.530536  8580 raft_consensus.cc:2393] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 20: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:21.531922  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:21.495772 15478 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:21.502367 15276 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.493486 15384 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.506357 15413 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:21.549778  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.550808 14505 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:21.498665 15376 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.544760  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.554497  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.555713  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.556990  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.559679  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.566895  8728 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:21.568809  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:21.571578 14505 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:21.572314 14505 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.573019 14505 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:21.574008  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:21.578287  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51533.1us
I20260709 13:04:21.591235 15193 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:21.508580 15464 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 13:04:21.579813  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.604423  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.579876  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.581171 15413 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.607605  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.587167  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.610433  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.610930  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.587831 15361 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:21.529201 15406 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.552203 14963 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:21.586263  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.605845  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:21.606243  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.606069 14505 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.555028 15478 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:21.612607  8733 raft_consensus.cc:2393] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:21.613499  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.576112 15401 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:21.611152  8732 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:21.620831  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.621127  8625 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:21.621382  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.566915 15324 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:21.622951  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:21.623816 15479 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:04:21.624409  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.639473  8726 raft_consensus.cc:2410] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:04:21.624667 15432 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.625743 15413 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:21.626216 15020 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.627084 15173 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:21.627456  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:21.628017 14893 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:21.625656 15476 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:21.628027 15384 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:21.633759 15407 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:21.622191  8732 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.636608  8698 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.642168  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.650745 14505 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.625612 15406 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.653756  8726 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:21.567233 15200 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.656914  8726 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 13:04:21.646279 15475 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.152s	user 0.001s	sys 0.001s
I20260709 13:04:21.662643  8726 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:04:21.642848 15464 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 13:04:21.666682 15376 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.669068  8726 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
W20260709 13:04:21.669354  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48743.1us
I20260709 13:04:21.660574  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:21.675793  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:21.677263 15415 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:21.679986 15432 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:21.662529 15020 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:04:21.674815 15478 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:21.682078 15477 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.127s	user 0.002s	sys 0.000s
I20260709 13:04:21.683544 14893 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:21.684005 14893 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:21.622108 14963 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.684453 14893 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.650553 15484 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:21.668318  8654 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.694061  8654 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.651253 15401 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.690420  8732 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:21.644248 15324 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.694839  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.695288  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:21.701987 15137 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.516s	user 0.004s	sys 0.008s
I20260709 13:04:21.701713 14688 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.702463  8730 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:21.703213 15324 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:21.704073 15287 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:21.705668  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.719581  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.693003 15050 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:21.700615 15401 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.707175 15300 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:21.708775  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.687273 15406 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:21.696249 15020 consensus_queue.cc:1175] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1dc41db601fe4b91aa6d461b504da0e7" responder_term: 17 status { last_received { term: 17 index: 9 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:04:21.722788  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.723884  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.699321 15381 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 13:04:21.728452  8659 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
W20260709 13:04:21.743317 15444 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260709 13:04:21.730398 15301 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:21.749966  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40408 (request call id 3071) took 1160 ms. Trace:
I20260709 13:04:21.749691  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2946) took 1222 ms. Trace:
I20260709 13:04:21.750437  8727 rpcz_store.cc:276] 0709 13:04:20.588581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.741630 (+153049us) service_pool.cc:224] Handling call
0709 13:04:21.749261 (+1007631us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b5400339854. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:21.749381 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37124480}
I20260709 13:04:21.740365 15401 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.750937  8725 rpcz_store.cc:276] 0709 13:04:20.527461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.527899 (+   438us) service_pool.cc:224] Handling call
0709 13:04:20.528283 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.767822 (+239539us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.749481 (+981659us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400203fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:21.749629 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":404748672}
I20260709 13:04:21.741194 14963 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:21.741417  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.743224  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:21.725095  8656 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:21.753607  8659 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:21.754225  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:21.755848  8655 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:21.756244  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.720238 15137 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:21.749701 15449 log.cc:927] Time spent T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.731s	user 0.003s	sys 0.000s
I20260709 13:04:21.760385  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.749922 14688 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:21.726747 15478 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:21.769209  8655 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:21.768193  8656 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:21.769750  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.770792  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.773226  8656 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:21.774910  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.775544  8725 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:21.775864  8725 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:21.777354  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.779083  8658 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:21.778590  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.776578  8725 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.784235  8725 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:21.784304  8658 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:21.790637  8732 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:21.791589 15436 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.782729 15406 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:21.792351  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.792783 15406 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.795042 15406 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.795747  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.796993  8732 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.808736  8732 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:21.810341  8654 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 13:04:21.810763  8654 pending_rounds.cc:85] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 8
I20260709 13:04:21.810971  8654 pending_rounds.cc:107] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 13:04:21.811182  8654 raft_consensus.cc:2889] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:21.811820  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3361) took 1256 ms. Trace:
I20260709 13:04:21.812100  8654 rpcz_store.cc:276] 0709 13:04:20.555131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.557657 (+  2526us) service_pool.cc:224] Handling call
0709 13:04:20.557931 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.740390 (+182459us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b54003ac120. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409b8ddc0 00007f1409c181c0 00007f1409c19231 00007f1409c15d8f 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c
0709 13:04:21.667943 (+927553us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b4c0039e680. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1407ee2a20 00007f1407ef3a29 00007f1407eb9644 00007f1407ee0f2d 00007f1407ee0b93 00007f1409c1118b 00007f1409c11cb5 00007f1409c0cf30 00007f1409c0e7de 00007f1409c18b49 00007f1409c19416
0709 13:04:21.811592 (+143649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.811781 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":157399168}
I20260709 13:04:21.813140  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:21.759802 15324 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.019s
W20260709 13:04:21.816269 15303 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 13:04:21.823318  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.823899  8727 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:21.795075 15471 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.453s	user 0.000s	sys 0.002s
I20260709 13:04:21.813759  8654 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:21.770885 15478 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:21.830641  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:21.805687 15488 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:21.841828 15488 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.842278 15488 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.838733  8650 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:21.836325  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:21.845672 15484 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:21.846257 15484 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:21.847185 15484 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.849865 15406 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:21.850872 15485 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:21.853477  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:21.853991 15444 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:21.855350 15478 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:21.855943 15444 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:21.856382 15478 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.856417 15444 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.854100 15324 consensus_queue.cc:1059] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:21.854923 15485 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:21.864096  8725 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:21.872373  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:21.880709  8725 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:21.875494  8650 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
W20260709 13:04:21.886356 15486 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 13:04:21.887534  8725 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:04:21.891324  8727 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:21.861142 15137 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:21.871134 15461 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:21.901655 15384 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:21.898007 15324 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.011s
I20260709 13:04:21.903173 15401 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.913964 15401 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.891243 15300 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 13:04:21.897374 15461 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:21.919265 15461 raft_consensus.cc:3060] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:21.930773  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.931864 15287 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:21.932315 15287 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:21.933115 15287 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.934247 15287 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:21.894897 15193 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:21.978186 15193 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:21.979130 15193 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:21.980242 15193 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.977540 15401 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:21.981812  8654 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:04:21.984549  8654 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:04:21.985530 15467 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:21.986347  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 406524us
I20260709 13:04:21.987149  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.988175 15467 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:21.988837  8549 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:21.989283 15460 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:21.992482  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:21.993927 15363 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:21.994360 15363 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:21.994743 15363 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.996969  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:21.999540  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3267) took 1131 ms. Trace:
I20260709 13:04:21.999940 15413 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.000504 15413 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.001946  8581 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 13:04:21.999949  8580 rpcz_store.cc:276] 0709 13:04:20.867777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.868180 (+   403us) service_pool.cc:224] Handling call
0709 13:04:21.999324 (+1131144us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b54003205d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:21.999461 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1468307200}
I20260709 13:04:22.005014  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:21.948809 15488 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.012966 14906 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 13:04:22.033488 15467 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:22.034114 15467 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:22.035053 15467 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.027519 15436 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.039723  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:22.039826 15436 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.040861  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:22.041762  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:22.041893  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53197.1us
I20260709 13:04:22.044123 15413 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.044596  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.042248 15436 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.042783 15415 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:22.045531 15414 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:22.045974 15415 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:22.046096 15414 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:22.046973 15415 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.047092 15414 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.043040 15432 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:22.049862 15432 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:22.043220 15400 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:22.059288  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:22.054180  8727 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:04:22.061816  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:22.055437  8725 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: [14.13-16.14]
W20260709 13:04:22.056914  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:22.064743  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.050845 15432 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.072516  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.076757  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.081640  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.011987 14893 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.069183 15453 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:22.092278 15460 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:22.092790 15361 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:04:22.067595 15413 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.075441 15467 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 13:04:22.093839 15413 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.093196 15453 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:22.096174 15413 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.098392  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.101025  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:22.105988  8727 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:22.107161 15413 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:22.108203  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2984) took 1327 ms. Trace:
I20260709 13:04:22.108594  8652 rpcz_store.cc:276] 0709 13:04:20.781000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.913667 (+132667us) service_pool.cc:224] Handling call
0709 13:04:22.108012 (+1194345us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001d1fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:22.108145 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284019456}
I20260709 13:04:22.109163  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.110867  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.078104 15364 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:22.079147 15474 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:22.115729  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:22.083316 15400 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:22.121803  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:22.113000 15453 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:22.128563  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:22.131408  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:22.080736 15436 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 13:04:22.132356  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.148898  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:22.114467 15474 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:22.138187  8659 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:22.147238  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.149255  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:22.153148  8650 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:22.118885 15407 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:22.133049  8730 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:22.162703  8652 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:22.164145  8730 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:04:22.161669  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:22.165187  8727 raft_consensus.cc:2368] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19.
I20260709 13:04:22.125835 15413 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:22.166649  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3079) took 1565 ms. Trace:
I20260709 13:04:22.171833  8728 rpcz_store.cc:276] 0709 13:04:20.600970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.832216 (+231246us) service_pool.cc:224] Handling call
0709 13:04:20.832473 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.567429 (+734956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.166455 (+599026us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b54004727d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:22.166592 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1688486144}
I20260709 13:04:22.174556  8625 leader_election.cc:400] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19.
I20260709 13:04:22.175094  8625 leader_election.cc:403] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:22.153997 15020 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:04:22.181010  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.154220 15436 consensus_queue.cc:1243] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:22.152415 15400 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.159333 14893 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.162107 15474 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:22.148813 15444 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.176589 15394 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:22.206463  8659 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:22.211123  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 2985) took 1400 ms. Trace:
I20260709 13:04:22.211617  8653 rpcz_store.cc:276] 0709 13:04:20.811198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.861340 (+ 50142us) service_pool.cc:224] Handling call
0709 13:04:22.210955 (+1349615us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b540031cc54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:22.211065 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1421446016}
I20260709 13:04:22.214239 15413 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:22.215761 15413 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.179776 15287 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 13:04:22.201088 15504 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.204784 15394 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:22.274724  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.200947 15467 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.207813 15478 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19.
W20260709 13:04:22.265588 15498 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 13:04:22.282223  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:22.283865  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:22.286003  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:22.285185  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:22.300423  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 1.49579e+06us
I20260709 13:04:22.287264  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:22.302093  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1214) took 2686 ms. Trace:
I20260709 13:04:22.290364  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:22.294988  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:21.332191  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:22.302551  9326 rpcz_store.cc:276] 0709 13:04:19.595158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.598773 (+  3615us) service_pool.cc:224] Handling call
0709 13:04:19.599125 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:19.599542 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:19.618386 (+ 18844us) log.cc:844] Serialized 14 byte log entry
0709 13:04:20.603486 (+985100us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b18004978c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:04:20.792084 (+188598us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:20.803822 (+ 11738us) log.cc:844] Serialized 26 byte log entry
0709 13:04:20.804013 (+   191us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:20.807090 (+  3077us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.710928 (+903838us) spinlock_profiling.cc:243] Waited 4.22 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:22.280855 (+569927us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b18004978c0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:04:22.281045 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.281124 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:22.281171 (+    47us) raft_consensus.cc:1685] finished
0709 13:04:22.281186 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:22.281324 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1471705,"prepare.queue_time_us":122890,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":1481677,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":12415,"spinlock_wait_cycles":26999040,"thread_start_us":190875,"threads_started":2,"wal-append.queue_time_us":1296710}
I20260709 13:04:22.306318  8575 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:22.208637 15414 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 13:04:22.203112 15436 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 13:04:22.199388 15400 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:22.314706  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 129562us
I20260709 13:04:22.276225 15488 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:22.320315  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1447) took 1178 ms. Trace:
W20260709 13:04:22.321774  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.351364  8575 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
W20260709 13:04:22.352056  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.326124 15364 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:22.353214  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.350854  9322 rpcz_store.cc:276] 0709 13:04:21.142128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.142288 (+   160us) service_pool.cc:224] Handling call
0709 13:04:21.142549 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.142791 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:21.143092 (+   301us) log.cc:844] Serialized 14 byte log entry
0709 13:04:22.319863 (+1176771us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:22.319925 (+    62us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:22.319978 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.320067 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:22.320249 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1176478,"prepare.queue_time_us":362,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":1177533,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100}
I20260709 13:04:22.326826 15068 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.327723 15397 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:22.328264 15311 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:22.330011 15280 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.314384 14948 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:22.379890 14948 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.380479 14948 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:22.387149 15397 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.331131 14688 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.305145 15467 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.332330 15363 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.334069 15253 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:22.341635  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:22.356066  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:22.357542  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.422469  9325 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:22.359210  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.359397  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.360844  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.361625  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.426090  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:22.367552  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1272) took 1645 ms. Trace:
I20260709 13:04:22.426961  8580 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:04:22.426883  8729 rpcz_store.cc:276] 0709 13:04:20.721754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.927874 (+206120us) service_pool.cc:224] Handling call
0709 13:04:20.986370 (+ 58496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:21.177635 (+191265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.367301 (+1189666us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b5400229f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:22.367492 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":417203968}
I20260709 13:04:22.370287  9329 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:04:22.277621 15504 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.429750  8582 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:22.430989 15504 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:22.348943 15497 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 13:04:22.369741 15454 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:22.374814  8578 raft_consensus.cc:1240] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:22.431933 15454 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:22.433003 15454 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.433521  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:22.434577 15454 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:22.435349  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:22.375825 15406 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:22.437600 15406 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:22.438774 15406 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.376453 15212 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:22.440160 15212 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:22.441288 15212 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.442620 15212 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:22.377251  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.444955  9331 raft_consensus.cc:2410] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:04:22.378720 15280 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.447113  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:22.447808  9329 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:22.448076 15280 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.448715  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.448892 15280 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.417040  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2734) took 1319 ms. Trace:
I20260709 13:04:22.453377  8575 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:22.453855  8581 rpcz_store.cc:276] 0709 13:04:21.097628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.196588 (+ 98960us) service_pool.cc:224] Handling call
0709 13:04:22.001685 (+805097us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:22.002374 (+   689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.416759 (+414385us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54003205d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:22.416955 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1850687232}
I20260709 13:04:22.457204  8581 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:22.437626  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:22.471547  8729 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:22.428757  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:22.451915  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.472808  8580 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:22.472944  9326 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:22.457492  8651 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:22.373075 15505 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:22.338443 15461 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.482584  8575 raft_consensus.cc:1240] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:22.482982  8650 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:22.476146  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:22.483834  8657 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:22.480595  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.373382 15068 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.487294  9331 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:22.449880 15253 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:22.476907 15505 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:22.466317 15362 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.506529  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:22.511562  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.511940  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.512374  9326 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:22.503226  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:22.516085 15287 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:22.518178  9325 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:22.520576  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.514737  9329 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:22.527843  9324 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:22.391742 15311 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.465006 15280 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.451696 14688 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.367096 15394 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.541723  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:22.529985  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:22.546267  9329 raft_consensus.cc:3060] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:22.546859  9327 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:22.519409 15362 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.520961  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:22.548554  9328 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 12: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:22.551218  8580 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:22.551769 15407 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:22.553089 14770 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.553781 14770 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.555230  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:22.556432 15407 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:22.556916 15407 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.557382 15407 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:22.525164  8657 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:22.557648 14770 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.485826 15173 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 13:04:22.518018 15401 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.518105 15414 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.525820 15361 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.535938 15444 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.536204 14963 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.586632 15276 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:22.587354 15276 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.589542 14963 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.592011 14963 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.593585 15068 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.567836 15397 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.596616  8622 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:22.559434 15506 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:22.594205 15068 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:22.603485  9329 raft_consensus.cc:2468] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:22.605518  8729 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:04:22.612505  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.614445 15510 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:22.640705  9329 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:22.575601 14893 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
W20260709 13:04:22.648965  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 129953us
I20260709 13:04:22.651355  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.651703  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.652493  9329 raft_consensus.cc:2376] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16. Re-sending same reply.
I20260709 13:04:22.607527 15506 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.598853 15303 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 13:04:22.602792 15463 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 13:04:22.687405  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:22.597540 15361 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.584429 15401 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.595347 15414 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.643071 15020 raft_consensus.cc:2804] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:22.618924 15068 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.698449 15464 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:22.681458 15400 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.684689 15488 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:22.583774 15505 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:22.684859 15506 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:22.686518 15397 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.689025 15511 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.640903 15510 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.709714 15363 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.714243 15467 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.716248 15464 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:22.725003 15461 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.763736 15505 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:22.763936 15397 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.799454 15397 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:22.768157 15511 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.822952 15511 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.825814 15510 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:22.828255 15506 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.831305 15506 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:22.884814  8578 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:04:22.886219  8578 pending_rounds.cc:85] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 7
I20260709 13:04:22.886693  8578 pending_rounds.cc:107] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:04:22.887154  8578 raft_consensus.cc:2889] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:22.606200  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:22.891115  8577 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:22.893090  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:22.894476  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 378578us
W20260709 13:04:22.699167  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 175918us
W20260709 13:04:22.902791  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:22.895215  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.696141  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:22.914294  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:22.916821  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 220601us
I20260709 13:04:22.919523  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:22.922896  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:22.924311  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:22.700018  9329 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.927693  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:22.932734  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:22.933494  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.942566  9329 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:22.944859  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.947068  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.948962  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:22.952611 14827 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:22.959895 14906 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:22.960201 15068 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:22.960999 14906 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:22.962136 14906 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.962545 14906 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:22.964211 15068 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:22.965945 15068 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:22.969135 15068 raft_consensus.cc:2764] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
I20260709 13:04:22.971947 15173 consensus_queue.cc:1059] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:22.973956 15193 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:22.718127 15020 raft_consensus.cc:697] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:22.953765 14827 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:22.764202 15276 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.094667 15303 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.095774 15303 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.096271 15363 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.097138 15363 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.097972 15510 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:23.100310 15363 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:23.107132 15529 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.282s	user 0.006s	sys 0.022s
I20260709 13:04:23.107014 15511 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:23.109370 15511 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:23.110615 15193 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:23.110713  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 482348us
I20260709 13:04:23.112366 15193 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:23.113626 15514 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.340s	user 0.016s	sys 0.010s
I20260709 13:04:23.116950  9322 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:04:23.117797 15477 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.479s	user 0.025s	sys 0.050s
I20260709 13:04:23.119328 15414 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.120541 15510 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.123100 15528 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.262s	user 0.000s	sys 0.010s
I20260709 13:04:23.125886 15361 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:23.130271 15527 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.264s	user 0.001s	sys 0.010s
I20260709 13:04:23.130640 15020 consensus_queue.cc:237] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.120584 14893 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.134686 14893 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.135473 15401 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.136466 15311 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.107000 15400 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.142068 15400 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.105563 15432 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.147336 15432 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:22.818197 15464 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.087146 14827 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.155020  9294 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.001s	sys 0.009s
W20260709 13:04:23.155521  9294 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.001s	sys 0.009s
I20260709 13:04:23.156041 14827 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:23.157147 15203 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.009s	sys 0.009s
I20260709 13:04:23.157562 15464 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:23.099563 14948 consensus_meta.cc:287] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.718s	user 0.003s	sys 0.013s
I20260709 13:04:23.161636 15538 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.161834 14948 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.162470 15538 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.164124 14948 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.164633 15538 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:23.162672 15432 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.166404 15538 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.167168 15538 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.170111 15538 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:23.157730 15203 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.009s	sys 0.009s
I20260709 13:04:23.169965 15510 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.172950 14948 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:23.173736 14948 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.176296 14948 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.177158 14948 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.177907 14948 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.194408 14906 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.137271 14893 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.190608 14948 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:23.206189  8656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.009s	sys 0.002s
W20260709 13:04:23.206636  8724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.005s	sys 0.005s
I20260709 13:04:23.209643 15522 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.210776 15539 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:23.211880 15413 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.212520 15276 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:23.214648  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 144484us
W20260709 13:04:23.215273 15212 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.742s	user 0.014s	sys 0.012s
I20260709 13:04:23.217790 14827 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:23.212778  8473 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 10 to 15, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:23.220526  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.204602  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.222260  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.224062 15400 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:23.234921  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.237669  8724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.006s	sys 0.006s
W20260709 13:04:23.238046  8656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.010s	sys 0.003s
I20260709 13:04:23.242522 15522 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.245689  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.246120 15474 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 1.028s	user 0.017s	sys 0.009s
I20260709 13:04:23.245857 15413 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.247285 15539 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.269747  8581 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:23.270576 15413 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.271366  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.272382  8581 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:23.281006 15276 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.281607  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 56883.8us
I20260709 13:04:23.270615 15522 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.299933 15539 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.302734  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:23.304095 15444 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:23.305101 15020 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:23.311326 15020 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:23.311789 15020 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:23.305065 15444 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.310398  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:23.276254 15474 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.314563  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3384) took 1593 ms. Trace:
I20260709 13:04:23.315969 15474 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.316377  8576 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:23.317116  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.317744 15549 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:23.318591  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.319191 15276 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.319773 15444 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.316959  8623 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:23.317432 14827 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.307322  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.323645  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.327747  9330 raft_consensus.cc:1275] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 13:04:23.328306  9330 pending_rounds.cc:85] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 14
W20260709 13:04:23.328403  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.328685  9330 pending_rounds.cc:107] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 13:04:23.329072  9330 raft_consensus.cc:2889] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:23.330086 14827 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.315110  8656 rpcz_store.cc:276] 0709 13:04:21.721641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.807135 (+ 85494us) service_pool.cc:224] Handling call
0709 13:04:21.807432 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:21.833875 (+ 26443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:21.833921 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:22.417791 (+583870us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b2c000b1fc8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:04:22.418319 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.281233 (+862914us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:23.284467 (+  3234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.290644 (+  6177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.290703 (+    59us) raft_consensus.cc:1685] finished
0709 13:04:23.290718 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.314480 (+ 23762us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26020,"raft.queue_time_us":30168,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":14995,"spinlock_wait_cycles":83842304,"thread_start_us":841935,"threads_started":1,"wal-append.queue_time_us":735071}
W20260709 13:04:23.333462  8696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.695s (SENT) [suppressed 13 similar messages]
I20260709 13:04:23.332870  8581 raft_consensus.cc:2410] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 13:04:23.335922 15280 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.339738 15253 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.340838  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.342919  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.343403 15324 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:23.343952 15324 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:23.345356  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.361879  8581 raft_consensus.cc:2410] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 13:04:23.345252 15324 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.363131  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.363824 15324 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:23.364483  8581 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:04:23.346310 14906 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.346333 15301 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:23.368515 15301 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:23.369319  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.369722 15301 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.370771  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.371510  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.371901  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:23.350006 15406 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.372968  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.358716 15444 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:23.377573 15444 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:23.378849 15444 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.370862 15301 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:23.372587  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:23.370877  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.385982  9322 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:04:23.391332  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.389114  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.381376 15362 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.403548  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.407747  9330 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:23.417626 15463 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.418684 15463 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:23.421941  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 105120us
I20260709 13:04:23.423377  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:23.424153  8584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1307) took 1121 ms. Trace:
I20260709 13:04:23.425815 15276 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.426580 15276 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:23.431013  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.432873  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3292) took 1125 ms. Trace:
I20260709 13:04:23.433360  8577 rpcz_store.cc:276] 0709 13:04:22.308136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.308882 (+   746us) service_pool.cc:224] Handling call
0709 13:04:22.309107 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.902794 (+593687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.432512 (+529718us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54001694d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:23.432801 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1335736704}
W20260709 13:04:23.434302  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.424600  8584 rpcz_store.cc:276] 0709 13:04:22.302541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.302685 (+   144us) service_pool.cc:224] Handling call
0709 13:04:22.303001 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.479638 (+176637us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.479681 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:22.479738 (+    57us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:22.479783 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.423682 (+943899us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54001680d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:23.423765 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.423948 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405689728}
W20260709 13:04:23.435536  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.434140 15362 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.439415 15362 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.408926 15253 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:23.443152  9329 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:23.428954 15276 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.451151  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.462847 15376 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.463721 15376 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.425854 15539 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:23.474303  8581 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:23.467666  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:23.476506  9329 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:04:23.479082  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 145674us
I20260709 13:04:23.480806  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:23.482359  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:23.487931  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.494562  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.495465  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.496268  9329 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:23.497256 15504 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:23.497622  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.497852 15504 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.498370 15504 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:23.499363  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.500473  8580 raft_consensus.cc:2410] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:04:23.501942  9330 raft_consensus.cc:2393] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:04:23.503355 15407 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.945s	user 0.013s	sys 0.008s
I20260709 13:04:23.503921 15407 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.494585 14906 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.506676 15207 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:23.506453 14906 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.512435  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:23.513078  8577 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.516640 15362 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.518302 14961 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:23.518651 15200 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:23.519006 14961 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.522114  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2965) took 1932 ms. Trace:
I20260709 13:04:23.521996  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.523108  8659 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:23.522709  8726 rpcz_store.cc:276] 0709 13:04:21.589284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.676141 (+ 86857us) service_pool.cc:224] Handling call
0709 13:04:21.676471 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:21.676856 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:21.699907 (+ 23051us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.884369 (+184462us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b1800259960. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b293f3 00007f1406b2b7c1 00007f1406b2b779 00007f1406b2b709 00007f1406b2b6d1 00007f1406b2a9cd 000000000038f354 000000000038f199 00007f1406b0c83a
0709 13:04:23.228132 (+1343763us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800259960. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:04:23.485448 (+257316us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.496248 (+ 10800us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.506014 (+  9766us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.506201 (+   187us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:23.506239 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.506375 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.521696 (+ 15321us) raft_consensus.cc:1685] finished
0709 13:04:23.521725 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.521909 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13970,"prepare.queue_time_us":49574,"prepare.run_cpu_time_us":162,"prepare.run_wall_time_us":3480,"raft.queue_time_us":33961,"raft.run_cpu_time_us":524,"raft.run_wall_time_us":10391,"spinlock_wait_cycles":36705536,"thread_start_us":171982,"threads_started":1,"wal-append.queue_time_us":1786255}
W20260709 13:04:23.524744  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 88825us
I20260709 13:04:23.526815  8653 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:23.533434 15071 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.510651 15463 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:23.534251 15071 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.538843 15394 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:23.534678  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:23.529938 15407 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.532162 14961 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.550832 15376 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:23.551309  9323 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:23.552160  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1235) took 1102 ms. Trace:
I20260709 13:04:23.552628  9323 rpcz_store.cc:276] 0709 13:04:22.449816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.450084 (+   268us) service_pool.cc:224] Handling call
0709 13:04:23.550654 (+1100570us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:04:23.550967 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.551807 (+   840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.552068 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17025280}
I20260709 13:04:23.553433 14906 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:23.553902  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:23.554267 14906 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:23.552167  8724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 2969) took 1804 ms. Trace:
I20260709 13:04:23.557731 14906 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.559670 15020 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.562783  8657 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:23.565052  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:23.566823  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.569034  8653 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 13:04:23.571624 14906 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.572464  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:23.573300  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.574170  8651 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:23.574522 15474 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:23.556885 15464 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:23.574981 15474 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:23.577463  8650 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:23.576176 15020 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.578750 15464 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:23.580276 15464 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.580787  8548 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:23.581794 15474 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:23.581759 15400 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:23.582561  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:23.583693  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.578282 15413 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.008s
W20260709 13:04:23.588052  8576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3286) took 1304 ms (client timeout 1695 ms). Trace:
W20260709 13:04:23.588533  8576 rpcz_store.cc:269] 0709 13:04:22.283535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.283645 (+   110us) service_pool.cc:224] Handling call
0709 13:04:23.587823 (+1304178us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003e7954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:23.587978 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2333016832}
I20260709 13:04:23.590369  8576 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
W20260709 13:04:23.592686  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.593807  8576 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:23.597354 15511 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.598330 15207 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:23.598596 15511 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.602738  8733 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:23.603446 15401 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.603987 15401 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.607187  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:23.600754 15474 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.570714  8724 rpcz_store.cc:276] 0709 13:04:21.747207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.748672 (+  1465us) service_pool.cc:224] Handling call
0709 13:04:21.749036 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:21.764335 (+ 15299us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:22.385453 (+621118us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b18004e6340. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:04:22.538567 (+153114us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.461990 (+923423us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.480010 (+ 18020us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.510498 (+ 30488us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.513554 (+  3056us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:23.513586 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.531429 (+ 17843us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.551406 (+ 19977us) raft_consensus.cc:1685] finished
0709 13:04:23.551426 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.551579 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25163,"prepare.queue_time_us":193029,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":79,"raft.queue_time_us":31665,"raft.run_cpu_time_us":620,"raft.run_wall_time_us":57975,"spinlock_wait_cycles":3335552,"thread_start_us":1033549,"threads_started":3,"wal-append.queue_time_us":616301}
I20260709 13:04:23.610199  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.614921  8659 raft_consensus.cc:2468] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:23.616062  8650 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:23.616972  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.607216 15413 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:23.612987 15509 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:23.615042 15200 consensus_queue.cc:1048] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.013s
I20260709 13:04:23.631915  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.614959 14906 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.636758  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.636960  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.639272  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1478) took 1127 ms. Trace:
I20260709 13:04:23.640031  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.639719  9324 rpcz_store.cc:276] 0709 13:04:22.511828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.519295 (+  7467us) service_pool.cc:224] Handling call
0709 13:04:22.527646 (+  8351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.532850 (+  5204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.638832 (+1105982us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x7b54002116d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:23.639095 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18449280}
I20260709 13:04:23.639796  9328 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:23.649184  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.649781  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:23.649964  9329 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
W20260709 13:04:23.651986  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 69606.8us
I20260709 13:04:23.652225  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:23.655404 14688 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.660655  8726 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:23.666924  9325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43664 (request call id 1234) took 1218 ms. Trace:
I20260709 13:04:23.665691 15504 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.665335  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1232) took 1217 ms. Trace:
I20260709 13:04:23.674201  9325 rpcz_store.cc:276] 0709 13:04:22.448159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.448403 (+   244us) service_pool.cc:224] Handling call
0709 13:04:23.666443 (+1218040us) spinlock_profiling.cc:243] Waited 68.7 ms on lock 0x7b5400867754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:23.666668 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":157966080}
I20260709 13:04:23.680142  9327 rpcz_store.cc:276] 0709 13:04:22.447378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.451143 (+  3765us) service_pool.cc:224] Handling call
0709 13:04:22.451470 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.547597 (+ 96127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.664653 (+1117056us) spinlock_profiling.cc:243] Waited 94.9 ms on lock 0x7b54007dc3d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:23.665052 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":218182528}
I20260709 13:04:23.689288  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.689868  8728 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:23.690402  8730 raft_consensus.cc:2410] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:23.670679 15401 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:23.693014 15474 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.693749 15303 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.693787  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:23.698098  8733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2956) took 2114 ms (client timeout 2771 ms). Trace:
W20260709 13:04:23.698597  8733 rpcz_store.cc:269] 0709 13:04:21.583347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.622638 (+ 39291us) service_pool.cc:224] Handling call
0709 13:04:23.696772 (+2074134us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b5400415354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:23.697998 (+  1226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4519216384}
W20260709 13:04:23.698133 15558 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 13:04:23.688608 14906 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.698884 15474 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:23.700538 15504 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.706599  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.713994  8728 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:23.714861  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:23.702548 15401 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.712602  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.714864  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:23.718943  8729 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:04:23.698931 15324 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 13:04:23.710974 15474 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:23.728134  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:23.724181 15504 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:23.733682  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.719959 15464 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:23.758558 15303 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.762645 15464 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:23.762746  8731 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:23.736239  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.738283  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.764535  8727 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:23.764798  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.753003 15453 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:04:23.766336  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:23.745762  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.780416  8730 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:23.748126  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2752) took 1323 ms. Trace:
I20260709 13:04:23.782104  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.723625  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.767161  8731 raft_consensus.cc:3060] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.767644 15464 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.762017 15504 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:23.747251  8656 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:23.762833  8582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3288) took 1477 ms. Trace:
W20260709 13:04:23.770138  8474 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 206) took 35410 ms (client timeout 15000 ms). Trace:
I20260709 13:04:23.762868 15303 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.797444  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:23.773274  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.800410 15303 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:23.774473  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:23.726046 15474 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.795177  8582 rpcz_store.cc:276] 0709 13:04:22.284828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.284944 (+   116us) service_pool.cc:224] Handling call
0709 13:04:23.762360 (+1477416us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b540002dd54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:23.762649 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":290173312}
I20260709 13:04:23.804203  8731 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
W20260709 13:04:23.802237  8474 rpcz_store.cc:269] 0709 13:03:48.359735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.360102 (+   367us) service_pool.cc:224] Handling call
0709 13:04:23.769842 (+35409740us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:16.569285 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:16.570009 (+   724us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:16.570057 (+    48us) write_op.cc:435] Acquired schema lock
0709 13:04:16.570081 (+    24us) tablet.cc:662] Decoding operations
0709 13:04:16.702325 (+132244us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:16.702416 (+    91us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:16.702443 (+    27us) tablet.cc:685] Acquiring locks for 19 operations
0709 13:04:16.704019 (+  1576us) tablet.cc:701] Row locks acquired
0709 13:04:16.704078 (+    59us) write_op.cc:260] PREPARE: finished
0709 13:04:16.704264 (+   186us) write_op.cc:270] Start()
0709 13:04:16.704429 (+   165us) write_op.cc:276] Timestamp: P: 1783602256704197 usec, L: 0
0709 13:04:16.704463 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 13:04:16.705378 (+   915us) log.cc:844] Serialized 7682 byte log entry
0709 13:04:17.309993 (+604615us) op_driver.cc:464] REPLICATION: finished
0709 13:04:17.526685 (+216692us) write_op.cc:301] APPLY: starting
0709 13:04:17.526845 (+   160us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:17.553935 (+ 27090us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:17.554055 (+   120us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:17.573118 (+ 19063us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:17.573539 (+   421us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0709 13:04:17.573609 (+    70us) write_op.cc:312] APPLY: finished
0709 13:04:17.574179 (+   570us) log.cc:844] Serialized 172 byte log entry
0709 13:04:17.575256 (+  1077us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:17.575462 (+   206us) write_op.cc:454] Released schema lock
0709 13:04:17.575543 (+    81us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:17.630934 (+ 55391us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:17.630976 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2544634,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":189139,"apply.run_cpu_time_us":28142,"apply.run_wall_time_us":123494,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83509,"num_ops":19,"prepare.queue_time_us":444953,"prepare.run_cpu_time_us":17050,"prepare.run_wall_time_us":213705,"raft.queue_time_us":376555,"raft.run_cpu_time_us":55664,"raft.run_wall_time_us":137369,"replication_time_us":605316,"thread_start_us":721654,"threads_started":4,"wal-append.queue_time_us":139526,"wal-append.run_cpu_time_us":35233,"wal-append.run_wall_time_us":1732694}]]}
W20260709 13:04:23.766584 15550 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.341s	user 0.003s	sys 0.002s
I20260709 13:04:23.764918  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.727314  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.765594  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.786707  8577 raft_consensus.cc:2468] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:23.741389  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3299) took 1177 ms. Trace:
I20260709 13:04:23.809589 15504 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:23.809787  8578 rpcz_store.cc:276] 0709 13:04:22.564278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.565624 (+  1346us) service_pool.cc:224] Handling call
0709 13:04:22.581905 (+ 16281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.888109 (+306204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.741093 (+852984us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54000b5754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:23.741326 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":693110016}
I20260709 13:04:23.810909 15538 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:23.812769 15362 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:23.813287 15362 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:23.814584 15362 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.823246 15362 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:23.825698 15324 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:23.826124  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.826508 15324 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:23.827183 15324 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:23.783542  8575 rpcz_store.cc:276] 0709 13:04:22.424635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.455916 (+ 31281us) service_pool.cc:224] Handling call
0709 13:04:22.456266 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.483286 (+ 27020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.747575 (+1264289us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b54002ca1d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:23.747893 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59833088}
I20260709 13:04:23.772104  8473 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 2 to 14, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 14 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:23.829602  8473 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 10 to 13, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 13 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:23.827350  8728 raft_consensus.cc:2410] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 13:04:23.832744  8473 catalog_manager.cc:5697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 7 to 13. New cstate: current_term: 13 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260709 13:04:23.837193  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.838320  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.838891  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:23.839545  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.840256  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.837311  8473 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 14 to 15, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 15 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:23.852632  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:23.853650  9327 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:23.868536 15301 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.870877 15212 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:23.869438 15301 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.875387 15538 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:23.876289 15538 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.881631 14688 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:23.881923 15136 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.882838 15136 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.790293  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 62294.7us
I20260709 13:04:23.886730 15136 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.906219  8698 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:23.908267  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.931603 15212 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:23.932847  9294 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:23.932591 15212 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.934109 15280 raft_consensus.cc:2804] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:23.934842 15280 raft_consensus.cc:697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:23.936086 15280 consensus_queue.cc:237] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.936798 15212 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.937764 15212 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.877050 15301 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.926031 15476 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.927143 15287 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:23.927983 15020 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.938331 15303 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:23.938918  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 66359us
I20260709 13:04:23.940271 15432 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:23.882747 14688 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:23.953675  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.896296  8473 catalog_manager.cc:5697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 4 to 14. New cstate: current_term: 14 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:23.958581  9327 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
W20260709 13:04:23.961683  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 122864us
I20260709 13:04:23.981281  8696 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:23.921025  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:23.971490  8724 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:23.978825  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.986579  8656 raft_consensus.cc:2410] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 13:04:23.987389  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.740367 15071 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:23.989598  8728 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:04:23.728250 15401 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:23.990267  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:23.991917  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.993163  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:23.954110 15453 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:23.994577  8726 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:04:23.960269 15394 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.997522  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:23.996693  8548 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:23.984925 15413 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:23.974318 15376 raft_consensus.cc:2804] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:23.969642 15200 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:04:24.015516  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 109439us
I20260709 13:04:23.993017 15303 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:23.961681 15538 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.024387  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:23.994778 15406 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:23.998111 14906 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:24.029635  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.035593  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.037914  8728 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:24.000576 15407 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:24.038245  8575 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:23.978161 15020 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.021503 15400 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:23.979965 15212 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.027006 15200 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:24.060238  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.061044  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.059415  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:24.017866 15394 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.069180  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:24.067992  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 69943.5us
I20260709 13:04:24.068423 15303 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.069132 15543 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:24.069350 14688 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:24.038688 15376 raft_consensus.cc:697] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:24.022354 15476 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.024770 15287 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.058066 15200 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:24.026413 15453 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.094499  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.096032 14770 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:24.111346 14770 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.096401  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.096740  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:24.114169 15560 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:04:24.114338  8653 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:24.096863  8726 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:24.097050  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.115674  8727 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:24.116254 15515 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:24.116478  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.116710  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3284) took 1834 ms. Trace:
I20260709 13:04:24.116907 15515 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.097429  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.117753  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.118055  9328 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.118468  8575 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:24.119256 15515 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.097651  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.121727  8581 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:24.097831  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.099079  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.123394  8580 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:24.099670  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.124447  8582 raft_consensus.cc:2393] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:24.100308  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:24.101079  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.106295  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:24.126438 15476 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.106729 15376 consensus_queue.cc:237] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.135109 15504 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.129768  8729 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:24.130656 15453 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.132051  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:24.134021  8728 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:24.125639  8473 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:24.055399 15538 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:24.039122 15432 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.146623  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:24.147388  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56680.9us
W20260709 13:04:24.159492  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 79537.8us
I20260709 13:04:24.040328 15413 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.161924  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:24.162549  8727 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:24.162768 15401 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:24.153175  8730 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:24.164522 15413 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.166414 15020 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.149927 15362 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:24.172610 15432 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.173842 15200 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.174607  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:24.175932 15324 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.176762  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:24.117077  8583 rpcz_store.cc:276] 0709 13:04:22.282779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.284152 (+  1373us) service_pool.cc:224] Handling call
0709 13:04:22.284638 (+   486us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:22.314697 (+ 30059us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.314735 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:22.315409 (+   674us) log.cc:844] Serialized 52 byte log entry
0709 13:04:23.638038 (+1322629us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b180002a9a0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:23.709935 (+ 71897us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:24.044122 (+334187us) log.cc:844] Serialized 14 byte log entry
0709 13:04:24.044631 (+   509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.044729 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:24.116267 (+ 71538us) raft_consensus.cc:1685] finished
0709 13:04:24.116376 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.116546 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32919,"prepare.queue_time_us":406574,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":21860,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":17712,"spinlock_wait_cycles":13887744,"thread_start_us":373169,"threads_started":2,"wal-append.queue_time_us":1632896}
I20260709 13:04:24.178264  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:24.178876  8583 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:04:24.175944 15413 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.182582 15413 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.183478  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.185855  8579 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:24.149555 14906 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.187537 14906 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.189543 14906 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:24.190032  8730 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260709 13:04:24.190428  8730 pending_rounds.cc:85] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 9
I20260709 13:04:24.190176  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.190662  8730 pending_rounds.cc:107] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 13:04:24.190924  8730 raft_consensus.cc:2889] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:24.191951 15394 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.192147  8727 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:24.183859 15476 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:24.086011 15543 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:24.157591  8656 raft_consensus.cc:2410] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 13:04:24.196197  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.193599  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.176366 15200 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.196516  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:24.066071 15406 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.209609  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:24.210561  8728 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:04:24.214936 15212 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.216202  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.216375  8576 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:04:24.203200 15362 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:24.217979 15362 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.218833 15324 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.220140 15362 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.220285  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.220588 15464 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.187565 15504 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.223299  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.224171  8727 raft_consensus.cc:2393] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:24.224833  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:24.076680 15474 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:24.229225  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.225255  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.230173  9330 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:24.234774  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.231280  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.237789  8732 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:24.238224  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:24.239004  8727 raft_consensus.cc:2368] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:24.182699 15301 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:04:24.245075  8724 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:04:24.160666 15453 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.209059 15303 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:24.256510  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.257859  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.258741  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.260097  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.208266 15543 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:24.217993 15565 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 13:04:24.202625 15476 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.245743  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.272305  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.246326  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.257434  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.278756  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:24.259518  8733 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:24.255836  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.280340  8582 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:24.281937  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.232650 15463 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.283517  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:24.286641  9325 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:24.289743  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.291231  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.305907 15136 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.306536 15136 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.307308  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.307829  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.308141  9329 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:04:24.309298  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.310011  8658 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:24.310216 15413 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.310302  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.310956  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.311668  8658 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:24.308748 15136 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.317070  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:24.321579 15453 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.323022 14770 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.325336  8473 catalog_manager.cc:5697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 3 to 16, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 16 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:24.327888  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.327721  8473 catalog_manager.cc:5697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 6 to 15. New cstate: current_term: 15 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:24.334568 15287 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.334945 15406 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.271777 15303 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.335904 15303 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.336736 15474 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.285620 15200 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.337822 15200 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.340435 15200 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:24.341612 15394 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:24.342442 15394 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:24.343695 15394 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.291249 15401 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 13:04:24.324328 15453 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.248481 14906 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:24.354980  8583 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:24.364681  9330 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 22.
I20260709 13:04:24.354763  8584 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:24.282995 15476 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:24.359313  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.234066 15538 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.375180  8728 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:24.365739 15400 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:24.388155  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.392547  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.393249  8657 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:24.378329 15207 raft_consensus.cc:2764] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:04:24.398408  9325 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
I20260709 13:04:24.375552 15324 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.412616  9329 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:24.360754 15463 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.407321 15020 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.407563 15400 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:24.413666  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 307507us
I20260709 13:04:24.415401  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:24.421918  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:24.422525 15071 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:24.423174 15071 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:24.423282 14948 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:24.426939  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.424562  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:24.379532 15509 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:24.417551  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 356114us
W20260709 13:04:24.420010  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.430783 15324 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.421656  8473 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 10 to 11, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 11 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:24.437131  8473 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 7 to 13. New cstate: current_term: 13 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:24.438122 15324 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.439229 15071 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.439682  8473 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 5 to 13, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 13 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:24.442898 15324 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.443194  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.445856 15324 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.446717  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:24.448139  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.449242  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.449309  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.450176  9323 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:24.450978  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.451615  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:24.452459  8651 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:24.445000  8473 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 15 to 17, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 17 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:24.454457  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.455322 15324 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.455430 15401 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:24.430106  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:24.455921 15401 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.456367 15401 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:24.456606 15573 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:24.457068 15573 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:24.457492 15573 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:24.455614  8651 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:24.458240 15538 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.458932 15464 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.459168  8651 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:24.456817  8473 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:24.462849 15463 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:24.466542 15573 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.467315  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.467607  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.467983  8584 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:24.404377 14906 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.468951 15509 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.469367  9297 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:24.469836 15020 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.471170 15509 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:24.471976 15020 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.473905 15280 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:24.474611 15287 consensus_queue.cc:1059] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:24.475147  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:24.475790 15287 raft_consensus.cc:3055] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:24.476167 15287 raft_consensus.cc:740] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:24.476703  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:24.477020 15287 consensus_queue.cc:260] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.478404 15287 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:24.479568  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.480527 15453 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:24.481654  8731 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:24.423943 14948 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.484995  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.486512  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.487702 15280 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:24.488440  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.488883 14906 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.489195  8729 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:24.488889  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:24.489704  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.490509  8657 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:24.491282  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:24.486786  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:24.495231  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:24.495891  8732 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.495604 15303 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.496865  8652 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:24.497690  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.499392  8548 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:24.500463 15400 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:24.501253 15400 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 22 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:24.501331 15543 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.502540 15400 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.503611 15543 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.505268  8732 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:24.536875  9329 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 13:04:24.537111  8651 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:04:24.538872  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:24.535305  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:24.544480  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.545439  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.550300  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.550926  8576 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:24.492937 14948 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:24.548408  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.557744  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.560562  9330 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:24.418151 15407 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:24.559293  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 76966.6us
I20260709 13:04:24.494421 15538 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:24.563808  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:24.509043 15136 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:24.564515  8657 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:24.564764  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:24.499562 15303 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.565627  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.567426  8581 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:24.562398  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.568466  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.571347  9328 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.461607  8473 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 9 to 11, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 11 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:24.563539  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.581641  8582 raft_consensus.cc:2368] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17.
I20260709 13:04:24.587059  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:24.582137  9322 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:24.593477  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:24.591476  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:24.598925  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 112230us
I20260709 13:04:24.599642  8655 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:04:24.600077  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:24.600922  8651 raft_consensus.cc:2368] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:24.536310 15020 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 13:04:24.614689  8550 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:24.561554 15572 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.594274 15540 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:24.642657 15540 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.643162 15540 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.644237 15569 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 13:04:24.536222 15574 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:24.646808 15574 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.623673 15573 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.649609 15413 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:24.651559 15504 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:24.652251 15504 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:24.622179  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:24.626816  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:24.656240  8732 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:24.616719 15303 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 13:04:24.627564  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:24.630638  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:24.590901 15543 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:24.638338  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:24.639307  8582 raft_consensus.cc:1275] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:24.640502  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.641752 14770 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:24.623330 15136 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.657900  8729 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:24.666026  8576 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:24.659052  8698 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:24.667850  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:24.615032 15212 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:24.667582  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:24.658723  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 384263us
I20260709 13:04:24.687321 15463 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.687961 15463 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.693683  9328 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:24.653483 15504 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.702396  8550 leader_election.cc:400] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
W20260709 13:04:24.708626  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 49031.9us
I20260709 13:04:24.665768 15569 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:24.659759 15543 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:24.666272 15324 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:24.671516 15207 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:24.715458  8651 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:04:24.695348 15574 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.691641 14906 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:24.662817 15136 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.671777 15511 consensus_queue.cc:1048] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 13:04:24.726351  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.729651  8657 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:24.710449 15280 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:24.730578 15280 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:24.731777 15280 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.719831 15543 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:24.721987 15207 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.743970 15207 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:24.746364 15463 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:24.752828 15476 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.755373  8653 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 22: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:24.706017 15407 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:24.736243  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.758213  8659 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:04:24.741516  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:24.769469  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 230686us
I20260709 13:04:24.770735  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.788518  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:24.791373  8622 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:24.794451  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 207983us
I20260709 13:04:24.725029 15413 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 13:04:24.797577  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:24.761946 15136 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:24.810232  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:24.816028  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.819216  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:24.801507 15303 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:24.822830  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:24.827432 15464 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:24.827713  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:24.827901 15464 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:24.828387 15464 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:24.828552 15509 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:24.833122  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:24.834079  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:24.815492 15301 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:24.834560  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:24.835055  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:24.840893  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 138674us
W20260709 13:04:24.854498  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 113121us
I20260709 13:04:24.861353  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.862893  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.864269  8658 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:24.825248 14770 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:24.870242  9325 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:24.824729 15579 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:24.901973  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:24.899289  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.855732 15476 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:24.858198 15400 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:24.899102  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.899758  9329 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:04:24.872900  9328 raft_consensus.cc:1275] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 13:04:24.900255  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.737013  8473 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 10 to 12, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:24.907902  8657 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 22: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:24.902585  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:24.919950  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:24.900892  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65059.5us
I20260709 13:04:24.922451  9327 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:24.854426 15301 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:24.898571  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63209us
I20260709 13:04:24.905210  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.816960 14906 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:04:24.873242 15509 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:24.911614  8696 leader_election.cc:400] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17.
I20260709 13:04:24.873638 15376 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:24.899644 15394 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.920847  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:24.905184 15578 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:24.924072  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:24.914906 15572 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:24.927927  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.927814  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.941180  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.941313  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.925889  9324 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:24.928107  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:24.915282 15400 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.930120  8696 leader_election.cc:403] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:04:24.944854  8578 raft_consensus.cc:2410] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 13:04:24.928907  9325 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:24.945470  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.946307  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.931475  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.927578 15574 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:24.913458 14770 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:24.947927  8580 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:24.953650  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 95982.7us
I20260709 13:04:24.968953  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:24.928524 15515 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:24.910722 15212 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 13:04:24.971796  9327 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:24.929265 15301 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:24.927752 15595 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.967516  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:24.997093  8623 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:24.933759 15509 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.942656 15578 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.983403  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:24.986589  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:24.942116 15394 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:24.989256  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:24.989928 15587 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 13:04:24.990290  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:24.938184 15476 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 13:04:24.992537 14906 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer a0d703ee18464475a290425976246555 (127.7.78.65:46779) with higher term. Message: Invalid argument: T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17.
I20260709 13:04:24.957701 15400 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:24.945703 15478 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:25.016878 15478 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:24.958460 14688 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:24.978807 15401 consensus_meta.cc:287] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.000s
I20260709 13:04:25.001952  9322 raft_consensus.cc:3060] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.978905 15515 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:24.999078  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:25.005571 15578 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:25.017269 15584 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 13:04:25.018280 14770 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.014719  8473 catalog_manager.cc:5697] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 8 to 14. New cstate: current_term: 14 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260709 13:04:25.016418 15583 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 13:04:24.986032 15569 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:25.011327  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 90489.1us
I20260709 13:04:25.030862 15478 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:24.990337 15362 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 13:04:25.001308 15212 consensus_queue.cc:1243] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer a0d703ee18464475a290425976246555 log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:04:25.001164 15509 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:25.034931  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:25.062672  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 92559.9us
I20260709 13:04:25.048236 15515 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:25.049736 14688 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.068444 15020 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:25.005905 15595 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.091964 14688 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.093304 15394 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:25.094931  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:25.049162 15401 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.073240 15212 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:25.118893 15212 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:25.123845 14948 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.000s
I20260709 13:04:25.124287 14948 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.061440 15599 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 13:04:25.121136 15212 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.142683  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:25.143430  8583 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:25.144611  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:25.145764 15596 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:25.148327 15596 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:25.148741 15596 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:25.150063 15595 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.154194  8577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3306) took 1062 ms. Trace:
I20260709 13:04:25.154664  8577 rpcz_store.cc:276] 0709 13:04:24.091996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.092146 (+   150us) service_pool.cc:224] Handling call
0709 13:04:25.153990 (+1061844us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b54002d78d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:25.154132 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2098098048}
I20260709 13:04:25.155834 15464 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.156472 15596 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:25.157053 15588 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
W20260709 13:04:25.158736 15585 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260709 13:04:25.158993 14948 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.139994 15574 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:25.161149  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 164333us
I20260709 13:04:25.162281  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:25.161378 15287 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:04:25.170945  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 173912us
W20260709 13:04:25.173099 15592 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 13:04:25.173269 15601 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:25.175165 15601 consensus_queue.cc:1048] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:25.177338  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:25.161202 15574 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.183692 15287 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:25.201663  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.202740  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.202466 15287 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.203084  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:25.201897 15601 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:25.203684  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:25.402168  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.403262  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.207276  8580 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:04:25.405901  9322 raft_consensus.cc:2468] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:25.402644  8650 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:25.299245  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 76808.5us
I20260709 13:04:25.408649  8657 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:25.402897  8654 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:25.164749 15212 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:04:25.428793  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 266596us
I20260709 13:04:25.403944  8655 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:25.432590  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:25.403604  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.168239 15474 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:25.439119  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.218324 15591 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.225s	user 0.004s	sys 0.000s
I20260709 13:04:25.444185  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:25.327661 15540 consensus_meta.cc:287] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.684s	user 0.012s	sys 0.000s
W20260709 13:04:25.472625  9297 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.020s	sys 0.014s
I20260709 13:04:25.443295  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.325011 15538 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:25.288638 15207 consensus_meta.cc:287] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.544s	user 0.011s	sys 0.000s
I20260709 13:04:25.430778 15212 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:25.209455 15515 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.223923 15301 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.451946  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:25.300222 15376 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.217332 15400 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.458489  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.461966  8650 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:25.299975 15071 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.184s
I20260709 13:04:25.380801 15478 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.219599 15543 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.469826 15595 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:25.445256  8651 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:25.470932  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:25.210906 15573 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:25.392261 15609 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:25.442008  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.474843  8726 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:25.279263 15589 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:25.409430 15601 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:25.475708  9297 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.020s	sys 0.014s
W20260709 13:04:25.547981  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 579051us
I20260709 13:04:25.391927 15464 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:25.478615  8696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.005s	sys 0.038s
W20260709 13:04:25.549996  8696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.005s	sys 0.039s
W20260709 13:04:25.550753  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 639321us
I20260709 13:04:25.551681  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:25.476770  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:25.479481  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:25.426754 15401 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:25.436048  8652 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:25.494513  8727 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:25.553958  8658 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:25.497009  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 64482.1us
I20260709 13:04:25.557145  9323 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
W20260709 13:04:25.481969 15303 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.012s	sys 0.032s
I20260709 13:04:25.351536 15578 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.435432 15579 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:25.479017 15287 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:25.464197 15596 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.502836  8657 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:25.513254  8653 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:25.516794  8575 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:25.496265 15212 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:25.522297  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:25.512154 15136 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:25.538638  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:25.517839 15614 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.573364  8580 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
W20260709 13:04:25.505802 15280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.016s	sys 0.035s
I20260709 13:04:25.496078 15207 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.548882  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:25.582705  9322 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:25.565667 15579 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.486476 15538 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.471891 15474 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.478075 15540 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.574745  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:25.544896  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:25.562987 15303 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.012s	sys 0.033s
I20260709 13:04:25.524952 15573 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:25.555300 15601 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:25.598363  8659 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:25.608384  8623 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:25.609515  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57958.1us
W20260709 13:04:25.585486 15280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.016s	sys 0.035s
I20260709 13:04:25.502458 15376 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.596933 15136 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.623558 15543 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.542621 15595 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.624518 15136 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.625190 15616 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:25.625991 15464 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:25.626261 15207 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:25.555822 15609 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:25.627389 15612 log.cc:927] Time spent T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 13:04:25.627533 15301 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.628911 15609 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:25.630427 15509 consensus_meta.cc:287] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.551s	user 0.012s	sys 0.000s
I20260709 13:04:25.630900 15509 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.604152 14688 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.631268  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:25.631796 14688 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.632375  8727 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:25.633000 15509 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:25.633580 15617 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 13:04:25.634016 14688 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:25.603704 15303 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:25.634447 15574 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:25.635588 15574 consensus_queue.cc:1243] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 17.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:04:25.635417 15303 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.636349 15574 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:25.637151 15569 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:25.637560 15569 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:25.638358 15569 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.639524 15569 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:25.642673  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 165986us
I20260709 13:04:25.643417  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:25.643716 15540 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:25.644196 15610 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:25.644675 15610 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:25.594931 15614 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.645025 15610 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:25.646127  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:25.647258  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:25.613657  9322 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:25.201498  8473 catalog_manager.cc:5697] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 8 to 13, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 13 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:25.621356 15621 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:25.623334 15476 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:25.634949 15579 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:25.603363 15596 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:04:25.728004  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 201461us
I20260709 13:04:25.729452  8579 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:25.710354  8726 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 15: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:04:25.720516 14770 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.030s	sys 0.093s
W20260709 13:04:25.734802  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 126473us
W20260709 13:04:25.788496  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 145144us
W20260709 13:04:25.788694  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 214028us
I20260709 13:04:25.788908  8732 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:25.606570 15599 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:25.666425 15136 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:04:25.840750  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.878585  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:25.792729  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:25.885340  8730 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:25.884552  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:25.873790  9326 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 13:04:25.868368  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 222280us
I20260709 13:04:25.885913  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:25.886739  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:25.875593  8623 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:25.886762  8725 raft_consensus.cc:2410] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 13:04:25.893929 15515 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:25.894821  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:25.895167  9324 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:25.895437  8582 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:25.628922 15573 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.896517 15573 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:25.898082  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:25.898990  9325 raft_consensus.cc:2393] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:25.898728  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:25.863955 15207 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:25.903802  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:25.864527 15401 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:25.907023  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:25.897270  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:25.908003  8580 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:25.874843 14770 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.031s	sys 0.093s
I20260709 13:04:25.902640 15394 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:25.876650 15596 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:25.911792 15394 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:25.912401 15596 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:25.898104  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:25.913877  9331 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:25.914616  8659 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:25.864326 15620 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:25.921514  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 1501) took 1046 ms. Trace:
I20260709 13:04:25.921933  9323 rpcz_store.cc:276] 0709 13:04:24.862413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.862578 (+   165us) service_pool.cc:224] Handling call
0709 13:04:25.909094 (+1046516us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54004dec54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:25.909203 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1594624896}
I20260709 13:04:25.912106  8657 raft_consensus.cc:1275] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:25.881469 15615 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.930699  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:25.945724  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:25.950541  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:25.907477 15324 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:25.954768 15394 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:25.941632 15616 raft_consensus.cc:2804] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:25.949779 15463 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:25.956651 15616 raft_consensus.cc:697] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:25.958056 15616 consensus_queue.cc:237] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.966500 15376 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:25.969993 15474 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:25.971910 15627 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
I20260709 13:04:25.910573 15207 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:25.978559  8730 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:25.978026  9331 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:25.980849  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2789) took 1045 ms. Trace:
I20260709 13:04:25.984934  8576 rpcz_store.cc:276] 0709 13:04:24.935341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.949213 (+ 13872us) service_pool.cc:224] Handling call
0709 13:04:24.949634 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:24.988050 (+ 38416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:24.988101 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:25.010056 (+ 21955us) log.cc:844] Serialized 26 byte log entry
0709 13:04:25.502698 (+492642us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:25.502756 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.502821 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:25.980545 (+477724us) raft_consensus.cc:1685] finished
0709 13:04:25.980576 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:25.980766 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19107,"raft.queue_time_us":32340,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":6316,"thread_start_us":437237,"threads_started":1,"wal-append.queue_time_us":539609}
I20260709 13:04:25.932377 15464 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:25.937917 15578 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:26.002116  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.016598  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:25.993456 15207 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.023427  8473 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 12 to 13, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 13 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:26.026342  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.027801  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:26.001287 15609 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 13:04:25.974900 15636 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:26.001477 15464 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:26.033885 15464 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:25.950848 15620 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:25.950217 15615 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.038245  9322 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:25.900132 15599 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:26.039377 15622 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 13:04:26.040367 15538 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:26.040733  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.023506  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41980 (request call id 3318) took 1094 ms. Trace:
I20260709 13:04:26.041556  8575 rpcz_store.cc:276] 0709 13:04:24.929250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.931982 (+  2732us) service_pool.cc:224] Handling call
0709 13:04:26.023305 (+1091323us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54003a3c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:26.023434 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1310218624}
I20260709 13:04:25.944768 14906 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.098s
I20260709 13:04:26.046236  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 3032) took 1131 ms. Trace:
I20260709 13:04:26.047011  8658 rpcz_store.cc:276] 0709 13:04:24.914976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.915468 (+   492us) service_pool.cc:224] Handling call
0709 13:04:26.045918 (+1130450us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540031cc54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:26.046118 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1395133184}
I20260709 13:04:26.049919 15623 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:26.051766  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:26.068454  9329 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:26.073540 15596 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.074738  9323 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:26.073223  8473 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 7 to 9, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER a0d703ee18464475a290425976246555 (127.7.78.65), VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66), VOTER 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67), VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 11 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:26.077991 15635 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:26.070907 15630 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 13:04:26.078781 15635 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.086215 15599 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:26.087209  9323 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:26.095036 15596 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:26.101186  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:26.102036  8731 raft_consensus.cc:606] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:26.102389  8731 raft_consensus.cc:614] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:26.102804  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.104686  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.105638  8575 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
W20260709 13:04:26.095644 15212 consensus_meta.cc:287] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.527s	user 0.008s	sys 0.003s
I20260709 13:04:26.107738 15212 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.050850 15605 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.158s
I20260709 13:04:26.110646 15212 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:26.097002 15615 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:26.111830 15595 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:26.113830  8577 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:26.123808  8732 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:26.125296 15614 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:26.099601 15601 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.127712 15601 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:26.128144  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41992 (request call id 2788) took 1193 ms. Trace:
I20260709 13:04:26.128598  8579 rpcz_store.cc:276] 0709 13:04:24.934993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.963528 (+ 28535us) service_pool.cc:224] Handling call
0709 13:04:26.127953 (+1164425us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b54002c97d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:26.128081 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1482761088}
I20260709 13:04:26.135277  8725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:26.136121  8725 raft_consensus.cc:606] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:04:26.136552  8725 raft_consensus.cc:614] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:26.138803  8580 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:26.140180  8575 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
W20260709 13:04:26.152011  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 276488us
I20260709 13:04:26.154640  8582 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:26.159166  8576 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 13:04:26.154331 15601 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:26.157008 15504 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.015s	sys 0.018s
I20260709 13:04:26.177583  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:26.180823  8577 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:26.176832  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:26.182478  8653 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:26.181239  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.160323 15413 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:26.185776  8658 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:26.176643 15504 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.016s	sys 0.018s
I20260709 13:04:26.169543 15635 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:26.154152 15207 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.219506  8725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:26.220173  8725 raft_consensus.cc:606] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:26.220548  8725 raft_consensus.cc:614] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:26.258814  8655 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 13:04:26.227029  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 311613us
I20260709 13:04:26.268507  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.323952  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:26.252591  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 368030us
I20260709 13:04:26.309772  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:26.436367  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.436810  8582 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:26.280782  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:26.277581  8725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:26.436106  8580 raft_consensus.cc:2393] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:26.439008  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.248152  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.378060  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.440168  8658 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:26.366402  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:26.267601  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.440958  8584 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:26.377370  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.446251  8576 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:26.387295  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3438) took 1056 ms. Trace:
I20260709 13:04:26.289361  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:26.449306  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 160022us
I20260709 13:04:26.394706  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.444808  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.452890  8730 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:26.447899  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:26.454573  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:26.455459  8729 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.438093  8725 raft_consensus.cc:606] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:26.408802  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:26.322098  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.458081  8579 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:26.246249  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.458797  8653 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:26.448915  8650 rpcz_store.cc:276] 0709 13:04:25.331208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.572240 (+241032us) service_pool.cc:224] Handling call
0709 13:04:25.597536 (+ 25296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:25.623142 (+ 25606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:25.623176 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:25.623225 (+    49us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:25.623263 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.387026 (+763763us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b54001d15d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:26.387078 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:26.387220 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":58409856}
W20260709 13:04:26.397907 15651 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.100s	user 0.000s	sys 0.003s
I20260709 13:04:26.462801  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.463198  8728 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:26.465071  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56408.2us
I20260709 13:04:26.466311  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:26.464527  8550 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:26.468043  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.469316  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:26.359481 15610 consensus_meta.cc:287] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.631s	user 0.008s	sys 0.006s
W20260709 13:04:26.320166 15569 consensus_meta.cc:287] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.680s	user 0.014s	sys 0.000s
I20260709 13:04:26.300104 15636 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:26.226151 15635 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.474383  8727 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:26.299676 15478 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:26.475499  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.483337  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.395080 15207 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:26.278589 15598 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:26.475492  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:26.433215 15362 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:26.451210 15573 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:26.451967 15609 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.228129 15623 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.484877  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:26.495201  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44431.4us
I20260709 13:04:26.210842 15601 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.216656 15413 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.265168 15620 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:26.364841 15400 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:26.479672  8581 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 13:04:26.493280  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:26.494216  9297 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.003s	sys 0.027s
I20260709 13:04:26.464529 15324 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:26.520947  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:26.524519  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.488613  8652 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.471935 15595 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:26.501876  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:26.276659 15464 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.352600 15453 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 13:04:26.353173 15568 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.136s
I20260709 13:04:26.483285 15504 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:26.505314  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:26.487810 15596 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:04:26.523545  9297 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.004s	sys 0.027s
I20260709 13:04:26.525529  8581 pending_rounds.cc:85] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 13
I20260709 13:04:26.554688  8581 pending_rounds.cc:107] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 13:04:26.555003  8581 raft_consensus.cc:2889] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:26.524170  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.493273 15598 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:26.519449  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:26.519413  8697 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:26.545657  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:26.480139 15610 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.558544  8727 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:26.549991  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:26.548894  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56146.2us
I20260709 13:04:26.556751  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:26.557096  9329 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:26.524334 15324 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:26.568351  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63098.6us
I20260709 13:04:26.493364 15635 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.500177 15362 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.569849 15640 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:26.576170  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:26.588124  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:26.577221  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:26.577594  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:26.588862  9329 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:26.548561 15636 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:26.591307  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:26.592586  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:26.532155 15595 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:26.569840  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:26.506349 15623 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.503985 15609 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.588244  9328 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:26.500425 15573 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.596289  9331 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 22: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:26.597079  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:26.612257  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43787.8us
I20260709 13:04:26.585089  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.560768 15596 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.527663 15509 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 13:04:26.582087 15324 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:26.582171 15599 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:26.572741 15478 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:26.620090  9326 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:26.569533 15653 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:26.595738 15605 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:26.644553  8652 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:26.632025  8731 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:26.573494 15643 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:26.614970 14906 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:26.618198 15596 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:26.637784 15573 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:26.637895 15478 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:26.606268 15595 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:26.639719  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:26.631168  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 129396us
I20260709 13:04:26.612870 15504 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:26.631582 15599 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:26.647419  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:26.648715  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.651805  9328 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:26.653360  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.654193  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.629236 15636 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:26.664206  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.683269  8656 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:26.664974  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.683357  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.684181  8659 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:26.685011  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:26.659776 15643 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.692211  8658 raft_consensus.cc:2368] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:04:26.693728  8724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea2142382a5d4439915e07c70837c688"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:26.694703  8724 raft_consensus.cc:606] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:26.601087 15640 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.686704  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.688408  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:26.696051  8732 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:26.679533 14906 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:26.704192  9325 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:26.665902 15574 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:26.655841 15598 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.707361 14906 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:26.708289 15635 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:26.666800 15605 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:26.709587 15605 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.711020  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:04:26.711091  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.639173 15642 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 13:04:26.682531 15599 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:26.679864 15207 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:04:26.727979 15207 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:26.728412 15207 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:26.729338 15207 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.704293  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:26.701037  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:26.730672 15657 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 13:04:26.721104  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:26.732318 15641 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.486s	user 0.001s	sys 0.001s
I20260709 13:04:26.713300  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:04:26.702975  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:26.740922  8657 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:26.795926 15636 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.799173 15504 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.801723 15478 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.987108  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:27.090155  9296 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.608s (SENT)
I20260709 13:04:27.126137  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:26.816425  8729 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:26.864432  8584 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
W20260709 13:04:27.072705  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 489601us
I20260709 13:04:27.128487  8698 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:26.804514  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 285163us
I20260709 13:04:27.124338  8652 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
W20260709 13:04:27.103327  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 529130us
I20260709 13:04:27.116786  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:26.996114  8659 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
W20260709 13:04:26.805833 15654 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 13:04:26.697134 15643 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:27.151427  9327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.021s	sys 0.020s
W20260709 13:04:27.152331  9327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.021s	sys 0.020s
I20260709 13:04:27.133498  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:27.019771 15596 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.136445  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:26.815323 15640 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.122380 15362 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:26.804579 15609 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.140635  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.108331 15464 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:26.981650 15303 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:27.095434  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.827756 15574 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.153460  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.173121  8659 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:27.122836 15413 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.155045  8656 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:27.064779 15509 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.152426  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.163363  8729 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:27.054042 15595 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:27.152901 15601 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:27.150770 15569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.001s	sys 0.018s
I20260709 13:04:27.122128 15207 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:27.166944  8547 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.005s	sys 0.043s
I20260709 13:04:27.167147 15663 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:27.220822  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.169934 15664 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:27.169838 15432 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:27.170748  9323 raft_consensus.cc:2368] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17.
I20260709 13:04:27.171864 15596 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:27.187577  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:27.218236  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48567.2us
I20260709 13:04:27.218842  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:27.222528 15324 consensus_meta.cc:287] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.583s	user 0.006s	sys 0.006s
I20260709 13:04:27.218453  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:27.222548  8547 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.006s	sys 0.043s
I20260709 13:04:27.168713 15400 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:27.223697  8656 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:27.238085  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:27.239550  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:27.240751  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:27.240969  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:27.241864  8582 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:27.242651 15659 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:27.243446 15659 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:27.245874  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:27.246641  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:27.247537  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:27.247540  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:27.249519  8656 raft_consensus.cc:2368] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:04:27.225679  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:27.166034 15610 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:27.175042 15464 raft_consensus.cc:993] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:04:27.228235  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.172297 15568 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:04:27.218487 15362 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:27.221957 15569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.002s	sys 0.018s
I20260709 13:04:27.232795  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:27.221176  8733 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.008s	sys 0.015s
W20260709 13:04:27.236603  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63446.4us
I20260709 13:04:27.260967  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:27.261732  8656 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:27.262009  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.184729 15303 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.191686 15599 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:26.724954 15662 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.550s
I20260709 13:04:27.222158 15611 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 13:04:27.220247 15453 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
W20260709 13:04:27.261006  8733 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.008s	sys 0.015s
I20260709 13:04:27.254621  8582 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:27.251379  9294 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:27.222764 15509 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.266919 15663 raft_consensus.cc:1081] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:04:27.223601 15665 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 13:04:27.249229 15324 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:27.252487 15664 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:27.258205 15640 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:27.271863  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:27.255563  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:27.281011  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.291702  8659 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:27.291946  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.276896 15599 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:27.260057 15601 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 13:04:27.244004 15659 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.293092  8652 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:27.267184 15635 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:27.258090 15413 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 13:04:27.311434  8725 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:27.298547  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:27.287949 15464 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:27.286685 15596 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:27.311371  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:27.319157  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 225223us
I20260709 13:04:27.296464 15400 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:27.300503 15504 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:27.286880 15668 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.029s
W20260709 13:04:27.296307 15540 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.008s	sys 0.024s
W20260709 13:04:27.334023  8473 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.014s	sys 0.040s
W20260709 13:04:27.342730  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 193823us
I20260709 13:04:27.313499 15464 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:27.348564  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 212217us
I20260709 13:04:27.351956  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.406414  8652 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 22: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:27.338790 15635 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:04:27.346511 15540 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.009s	sys 0.024s
I20260709 13:04:27.419785  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:27.419773 15635 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:27.421597 15635 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.433907  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.434948 15324 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:27.436579 15324 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:27.340248 15596 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:27.405814  8473 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.015s	sys 0.040s
I20260709 13:04:27.438882  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:27.448899  8652 raft_consensus.cc:2393] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:27.450224 15509 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.449818 15464 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:27.451586 15464 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:27.450281  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.407902 15574 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.469838  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.471096  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.471679  8652 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 10.
I20260709 13:04:27.475664  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.459331 15324 raft_consensus.cc:993] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:04:27.478523  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:27.481671  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:27.487210 15610 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:27.487859 15610 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.493639  8656 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:27.495323  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.496961  8656 raft_consensus.cc:2410] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 13:04:27.498008  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53385.1us
I20260709 13:04:27.498109  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.499150  8656 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:27.500272  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.501145  8656 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:27.502058  8729 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:27.451442 15413 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:04:27.517683  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:27.470475 15662 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:27.521775 15662 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.524737 15662 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.463299 15595 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:27.473066 15504 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:27.440171 15599 consensus_queue.cc:1048] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 13:04:27.513919  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:27.538376  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:27.535488  8729 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:27.525333 15662 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:27.552428  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:27.554036  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:27.554193  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 304540us
I20260709 13:04:27.554899  8728 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:27.559795  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.561105  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:27.580976  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 284901us
W20260709 13:04:27.613188  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57298.8us
I20260709 13:04:27.645663  8656 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:27.645813  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:27.647619  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:27.649065  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:27.658795  8549 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.783s (SENT)
I20260709 13:04:27.659454  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.448024 15664 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.670158  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:27.624249 15676 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 13:04:27.672446  8582 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:27.680629  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:27.682013  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:27.682607  8582 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:27.685488  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 387007us
I20260709 13:04:27.670904  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.688066  8650 raft_consensus.cc:2368] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18.
I20260709 13:04:27.536240 15596 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:27.675036  8549 leader_election.cc:336] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.807s (SENT)
I20260709 13:04:27.686420  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:27.686718  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.696857  8656 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 22: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:27.696903  8549 leader_election.cc:336] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.894s (SENT)
I20260709 13:04:27.698644  8582 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:27.699679  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:27.676834  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:27.701249  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.670086 15504 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.650352 15601 consensus_queue.cc:1243] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:04:27.709810  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1184) took 1043 ms. Trace:
I20260709 13:04:27.685118 15303 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.700318  8582 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 10.
W20260709 13:04:27.708518  8549 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.956s (SENT)
I20260709 13:04:27.722460  8656 rpcz_store.cc:276] 0709 13:04:26.666042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.686474 (+1020432us) service_pool.cc:224] Handling call
0709 13:04:27.709741 (+ 23267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.722882  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.671684 15661 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:27.678130 15666 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:27.691447 15679 raft_consensus.cc:1081] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:04:27.610112 15662 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.755125  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:27.756273  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 257445us
I20260709 13:04:27.729120 15504 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:27.753002 15678 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.652240 15643 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:27.757727  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:27.751623  8549 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.164s (SENT)
I20260709 13:04:27.707891 15595 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:27.729826  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.766333  8656 raft_consensus.cc:2410] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 13:04:27.757448  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:27.719240 15659 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:27.719697 15610 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:27.769953  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 3043) took 1096 ms. Trace:
I20260709 13:04:27.786237  8656 rpcz_store.cc:276] 0709 13:04:26.673757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.729553 (+1055796us) service_pool.cc:224] Handling call
0709 13:04:27.769885 (+ 40332us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.770165  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.787410  8582 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:27.787451  8656 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:27.788618  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3034) took 1122 ms. Trace:
I20260709 13:04:27.788941  8656 rpcz_store.cc:276] 0709 13:04:26.667368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.786980 (+1119612us) service_pool.cc:224] Handling call
0709 13:04:27.787203 (+   223us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.788413 (+  1210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.788572 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.792332  8656 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:04:27.760375 15568 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:27.757378 15601 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:27.830219  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 138897us
I20260709 13:04:27.769081 15540 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.803076  8698 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:27.979224  9296 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.901s (SENT)
I20260709 13:04:27.979890  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:27.981137  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3038) took 1308 ms. Trace:
I20260709 13:04:27.972990  8581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3328) took 1993 ms. Trace:
W20260709 13:04:27.980103  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 277129us
I20260709 13:04:27.982183  8581 rpcz_store.cc:276] 0709 13:04:25.979468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.979750 (+   282us) service_pool.cc:224] Handling call
0709 13:04:25.980119 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:26.555417 (+575298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.972722 (+1417305us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54002d78d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:27.972889 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147807488}
I20260709 13:04:27.995828  8656 rpcz_store.cc:276] 0709 13:04:26.672551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.791525 (+1118974us) service_pool.cc:224] Handling call
0709 13:04:27.791923 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.980542 (+188619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.981057 (+   515us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.798487 15509 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:27.822813 15595 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.976027 15324 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:27.754274 15661 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.982379 15400 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:28.031100  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 228103us
I20260709 13:04:28.013507  8656 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:28.013676  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:27.982834 15362 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:28.015645 15643 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:28.015393 15691 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 13:04:28.019809 15648 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.722s	user 0.008s	sys 0.013s
W20260709 13:04:28.020787 15683 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.222s	user 0.000s	sys 0.000s
I20260709 13:04:28.008759 15509 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:28.012341 15595 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.044761  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3041) took 1371 ms. Trace:
W20260709 13:04:28.049718  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.769656 15678 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:27.998867 15687 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:28.038692 15674 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.523s	user 0.000s	sys 0.001s
I20260709 13:04:28.093508  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:27.998061 15540 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.078751  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.073904  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.055745 15601 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:28.050047 14906 raft_consensus.cc:2804] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:28.040553 15303 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 13:04:28.109856  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.129191  9328 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:04:28.098296  8656 rpcz_store.cc:276] 0709 13:04:26.673263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.997261 (+1323998us) service_pool.cc:224] Handling call
0709 13:04:28.011640 (+ 14379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.044520 (+ 32880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.044704 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.057588 15574 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:28.119657  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.134203  8650 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:28.120471 15595 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:28.122754  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.124640 15614 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:28.125727 15681 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:28.131152  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.027045 15682 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.091s
I20260709 13:04:28.161367  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.162149  8655 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:28.135610  8653 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:28.135677  8549 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.582s (SENT)
I20260709 13:04:28.161371  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.162992  8549 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.155567 15695 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:28.156339 15662 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.097352 15509 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:28.157111  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 3007) took 1560 ms. Trace:
I20260709 13:04:28.101982 15705 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:28.128691 14906 raft_consensus.cc:697] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:28.157573 15574 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:28.158491 15684 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 13:04:28.097790 15413 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:28.156342  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1345) took 1802 ms. Trace:
I20260709 13:04:28.163861  8656 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:28.168008  8650 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:28.166206  8625 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:28.175262  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 175072us
W20260709 13:04:28.179867  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44229.4us
I20260709 13:04:28.158020 15540 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.181141  8725 rpcz_store.cc:276] 0709 13:04:26.596582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.596742 (+   160us) service_pool.cc:224] Handling call
0709 13:04:28.156926 (+1560184us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b54000e9a54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:28.157054 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1599042304}
I20260709 13:04:28.196816  8625 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:28.159250 15687 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:28.198558 15687 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.161312 15691 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:28.209157  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1505) took 1500 ms. Trace:
W20260709 13:04:28.202729  9296 leader_election.cc:336] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.192s (SENT)
I20260709 13:04:28.227820  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.228317  8625 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:28.115940 15478 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 13:04:28.227692  9322 rpcz_store.cc:276] 0709 13:04:26.708697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.116404 (+407707us) service_pool.cc:224] Handling call
0709 13:04:27.116825 (+   421us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:27.133285 (+ 16460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:27.133315 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:27.161283 (+ 27968us) log.cc:844] Serialized 52 byte log entry
0709 13:04:27.670577 (+509294us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:27.670622 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.670719 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:28.208895 (+538176us) raft_consensus.cc:1685] finished
0709 13:04:28.208918 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.209091 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5402,"raft.queue_time_us":8378,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":7538,"thread_start_us":499310,"threads_started":1,"wal-append.queue_time_us":1034955}
I20260709 13:04:28.193310  8575 rpcz_store.cc:276] 0709 13:04:26.354326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.475928 (+121602us) service_pool.cc:224] Handling call
0709 13:04:26.476195 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:26.654633 (+178438us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:26.665475 (+ 10842us) log.cc:844] Serialized 14 byte log entry
0709 13:04:27.173082 (+507607us) log.cc:844] Serialized 14 byte log entry
0709 13:04:27.173180 (+    98us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:27.173222 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:27.173264 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.156130 (+982866us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54000f96d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:28.156176 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.156285 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8636,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":71,"raft.queue_time_us":10133,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":13363,"spinlock_wait_cycles":377100032,"thread_start_us":448665,"threads_started":1,"wal-append.queue_time_us":568169}
W20260709 13:04:28.232092  9296 leader_election.cc:336] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.239s (SENT)
I20260709 13:04:28.242205  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.232671 15690 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:28.183887 15609 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:28.248056  8656 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
W20260709 13:04:28.249902  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 171244us
I20260709 13:04:28.228221  8653 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
W20260709 13:04:28.255468  9296 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.045s (SENT)
I20260709 13:04:28.257737  9296 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.266136  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.257530  8622 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.267287  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.267968  8658 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:28.267460  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.268847  8650 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 12.
I20260709 13:04:28.050433 15400 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.273197  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.277734  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.211576 15324 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:28.198758 15504 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:28.283151  8658 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:28.284425  8653 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:28.200001 15614 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:28.284484  8650 raft_consensus.cc:2368] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:04:28.284648  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.289019  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:28.293861  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.302210  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:28.311419  8725 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:28.201042 15687 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:28.169456 15681 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:28.305276  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47850.1us
I20260709 13:04:28.245611 15700 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:04:28.343389  8625 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.875s (SENT) [suppressed 39 similar messages]
I20260709 13:04:28.253707 15601 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.313643  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.267874 15705 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:28.277982 15538 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:28.341635  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.266821 15362 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:28.312427  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:28.311233 15686 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 13:04:28.352124 15601 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.345034 15543 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:28.266574 15691 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:28.349248  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.309412 15704 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:04:28.190275 15695 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.352577 15690 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:28.360069 15695 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.352903  8728 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:28.360697 15623 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.309132 15685 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.354279 15614 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 13:04:28.361783 15685 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.356050  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1506) took 1647 ms. Trace:
I20260709 13:04:28.363240  9331 rpcz_store.cc:276] 0709 13:04:26.708999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.250528 (+541529us) service_pool.cc:224] Handling call
0709 13:04:27.263167 (+ 12639us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:27.263558 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:27.263594 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:27.267654 (+  4060us) log.cc:844] Serialized 52 byte log entry
0709 13:04:27.645062 (+377408us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:27.646572 (+  1510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.646650 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:28.355794 (+709144us) raft_consensus.cc:1685] finished
0709 13:04:28.355818 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.355989 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":94,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":2448,"thread_start_us":352347,"threads_started":1,"wal-append.queue_time_us":840593}
I20260709 13:04:28.357378 15691 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.365005 15303 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:28.361114 15700 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.366371 15646 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:28.366266 15700 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.366832 15646 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:28.367723 15646 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.368606 15700 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.354158 15682 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:28.358080  9296 leader_election.cc:400] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18.
I20260709 13:04:28.369527  9296 leader_election.cc:403] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:28.370674  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.372457  9296 leader_election.cc:400] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:04:28.373355 15538 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18.
I20260709 13:04:28.358359 15704 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:28.375582 15704 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:04:28.378784 15647 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 2.078s	user 0.012s	sys 0.038s
I20260709 13:04:28.380052  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3148) took 2489 ms. Trace:
I20260709 13:04:28.380436  8726 rpcz_store.cc:276] 0709 13:04:25.890488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.890782 (+   294us) service_pool.cc:224] Handling call
0709 13:04:25.891066 (+   284us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:25.904593 (+ 13527us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:25.904622 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:25.934291 (+ 29669us) log.cc:844] Serialized 26 byte log entry
0709 13:04:26.203913 (+269622us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:26.203958 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.204024 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:26.275237 (+ 71213us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b480018c258. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:04:28.350027 (+2074790us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:28.379825 (+ 29798us) raft_consensus.cc:1685] finished
0709 13:04:28.379844 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.379974 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13136,"raft.queue_time_us":15321,"raft.run_cpu_time_us":3807,"raft.run_wall_time_us":359541,"spinlock_wait_cycles":47943680,"thread_start_us":230112,"threads_started":1,"wal-append.queue_time_us":286984}
I20260709 13:04:28.382438 15678 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.384371 15574 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.388033  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.393106  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.343173  8658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:04:28.394142  8658 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:04:28.394536  8658 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:28.350821  8650 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea2142382a5d4439915e07c70837c688"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:04:28.397095  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.397516  8650 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:04:28.398017  8650 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:28.344091 14906 consensus_queue.cc:237] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.403769 15400 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:28.405857 15685 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:28.406302 15324 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:28.406895 15324 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.407747  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.408929 15324 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:28.409765 15679 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:28.409916 15663 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:28.411049 15478 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:28.411656  9331 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:28.412343  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:28.412592  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:28.435272  8727 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:28.413746  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:28.436259  8726 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:28.415470  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.437556  9328 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 14.
I20260709 13:04:28.369231 15646 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:28.418437  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 220369us
I20260709 13:04:28.419257  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.419559  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.443121  9325 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:28.438799  8653 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
W20260709 13:04:28.444860  8549 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.671s (SENT)
W20260709 13:04:28.446226  8549 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.714s (SENT)
I20260709 13:04:28.446792  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.435853  8650 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.457849  8650 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.459919  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:28.460304  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 295145us
I20260709 13:04:28.460635  8733 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:28.461362  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.460948  8733 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:28.441336 15478 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.462363  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:28.465088  8623 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Timed out: RequestConsensusVote RPC to 127.7.78.68:33133 timed out after 1.875s (SENT)
I20260709 13:04:28.458323  8625 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.448817 15538 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:28.465634  8698 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.470489  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:28.473074  8698 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.474234  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.444844 15601 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:28.469583  8549 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.729s (SENT)
W20260709 13:04:28.480288  8549 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.853s (SENT)
W20260709 13:04:28.481606  8549 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.906s (SENT)
W20260709 13:04:28.464864 15207 consensus_meta.cc:287] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.241s	user 0.015s	sys 0.001s
I20260709 13:04:28.470309  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.449122 15574 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:28.525601  8732 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:04:28.471529  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.440459  8658 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.484913  8655 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:28.476090  8733 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.488528  8653 raft_consensus.cc:2410] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
W20260709 13:04:28.520040  9296 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.062s (SENT)
I20260709 13:04:28.482383 15324 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:28.468094 15478 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:28.454701 15609 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.442139 15679 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.467407 15702 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:04:28.534896  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 63441.6us
I20260709 13:04:28.529502  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.543007  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.544291  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.471488 15682 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:28.482993 15668 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:28.545099  8733 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:28.485548 15568 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:28.471243 15688 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:28.559188  8658 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:28.442324 15663 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.482419 15710 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:28.552104  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.547577 15362 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:28.554152  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:28.534647 15020 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:28.526507 15464 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.535141 15685 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:28.482614 15678 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:28.594095 15543 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:28.594394 15678 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:28.572713 15695 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:04:28.595291 15678 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.598352  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:28.598731 15679 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.599339  9330 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:28.599267 15704 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.599956 15464 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:28.548646  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 142025us
I20260709 13:04:28.564252  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:28.568359  8653 raft_consensus.cc:2368] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:04:28.601462  8727 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:28.549281 15504 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:28.555784 15324 raft_consensus.cc:697] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:28.557258 15568 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:28.551424 15207 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:28.578169  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.632457  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:28.587378 15663 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.553659 15702 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:28.550412 15574 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.567188 15710 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.588552  9326 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1499) took 1881 ms. Trace:
I20260709 13:04:28.549940 15609 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.560824 15682 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:28.601048 15685 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.593726 15020 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.556722  8726 raft_consensus.cc:1240] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:28.601909  8727 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:28.653688  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.598645  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:28.602766  8697 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:28.655529  8731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 2996) took 2213 ms (client timeout 1875 ms). Trace:
I20260709 13:04:28.562783  8655 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:28.655534  8725 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:28.607530  8733 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:28.639816  9326 rpcz_store.cc:276] 0709 13:04:26.706731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.729837 (+ 23106us) service_pool.cc:224] Handling call
0709 13:04:28.104144 (+1374307us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:04:28.104355 (+   211us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:28.134465 (+ 30110us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.134512 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:28.193517 (+ 59005us) log.cc:844] Serialized 52 byte log entry
0709 13:04:28.447406 (+253889us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:28.447470 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.447749 (+   279us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:28.588249 (+140500us) raft_consensus.cc:1685] finished
0709 13:04:28.588269 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.588491 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23399,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":25441024,"thread_start_us":141192,"threads_started":1,"wal-append.queue_time_us":324036}
W20260709 13:04:28.621199  9296 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.490s (SENT)
I20260709 13:04:28.657572  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.657732  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:28.575634 15668 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:28.608312  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 116789us
I20260709 13:04:28.635210 15568 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:28.635602  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1280) took 1009 ms. Trace:
I20260709 13:04:28.654289  8727 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:28.656026  8731 rpcz_store.cc:269] 0709 13:04:26.438262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.438683 (+   421us) service_pool.cc:224] Handling call
0709 13:04:28.651162 (+2212479us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400416254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:28.651285 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":387328256}
I20260709 13:04:28.640055 15690 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:04:28.665699  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62992.5us
I20260709 13:04:28.672765  8727 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:28.647055 15614 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:28.677074  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55961.5us
W20260709 13:04:28.694162  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:28.692937  9296 leader_election.cc:400] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:04:28.695216  9296 leader_election.cc:403] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 13:04:28.660351  8625 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.636340 15509 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.672556  9329 rpcz_store.cc:276] 0709 13:04:27.626813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.158238 (+531425us) service_pool.cc:224] Handling call
0709 13:04:28.158631 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.158834 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:28.463744 (+304910us) log.cc:844] Serialized 14 byte log entry
0709 13:04:28.635064 (+171320us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:28.635155 (+    91us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:28.635226 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.635340 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.635520 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12104,"prepare.queue_time_us":195627,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":5012,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":6487,"thread_start_us":473242,"threads_started":2}
I20260709 13:04:28.650859 15710 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:28.704996  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:28.701670  8549 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.951s (SENT)
I20260709 13:04:28.701740  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:28.705792  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.654337 15685 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:28.675077 15687 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:28.693580 15543 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:28.634469 15504 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.674829 15720 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:04:28.713331  8549 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.969s (SENT)
I20260709 13:04:28.639654 15207 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.714967  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:28.720148  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:28.707381 15691 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:28.713081 15687 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.739545  8653 raft_consensus.cc:1240] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:04:28.740103  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.717157 15688 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:28.700021 15614 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.746330  8727 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:28.719859 15720 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:28.753319  8653 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: [17.12-18.13]
I20260709 13:04:28.750677 15704 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.715344 15605 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 13:04:28.742622 15687 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:28.759858 15718 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 13:04:28.766624  8651 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:04:28.751786 15666 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:28.740921 15596 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 14. Current term is 15.
I20260709 13:04:28.740053 15719 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:28.778076  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.779246  9322 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:28.759888  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:28.777171  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:28.742352 15691 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:28.713436 15324 consensus_queue.cc:237] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:28.794021 15731 log.cc:927] Time spent T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:04:28.760110 15720 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:28.781373 15719 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:28.821476  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 49751.3us
I20260709 13:04:28.719357 15702 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.748458 15682 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.832048  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:28.855180  9324 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:28.795763 15478 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:28.820752 15719 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:28.752591 15668 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:28.916298  9324 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1500) took 2153 ms (client timeout 1637 ms). Trace:
W20260709 13:04:28.938308  9324 rpcz_store.cc:269] 0709 13:04:26.707140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.710655 (+  3515us) service_pool.cc:224] Handling call
0709 13:04:28.831898 (+2121243us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800003c98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1e9c 00007f1406af1a5b 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebdc0a0 00007f140ebdb984 00007f14081566d4 00007f1408156662
0709 13:04:28.860047 (+ 28149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24504448}
W20260709 13:04:28.941628 15699 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 13:04:28.941929 15737 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:28.942490 15737 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:28.944644 15737 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:28.946100 15207 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.946403 15020 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:28.857537 15609 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:28.914714  8657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 3039) took 2187 ms (client timeout 1783 ms). Trace:
W20260709 13:04:28.948726  8657 rpcz_store.cc:269] 0709 13:04:26.672795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.687536 (+ 14741us) service_pool.cc:224] Handling call
0709 13:04:28.832554 (+2145018us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b540018fbd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:28.859902 (+ 27348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23816576}
I20260709 13:04:28.949702  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:28.953884  8727 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:28.957165  8654 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:28.958694  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:28.955763 15020 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:28.966101  8727 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:28.961141 15020 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:28.990744  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:28.980531 15635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.551s	user 0.007s	sys 0.000s
I20260709 13:04:28.963337 15509 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:28.953161 15646 consensus_queue.cc:1059] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 13:04:29.006738  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:29.010360 15727 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:29.011801  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.012452  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.013015  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.013685  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.013847  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:28.916855 15691 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.022254  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.022920  8582 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:28.995146 15678 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.027539  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:29.028935  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.003475 15662 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:29.034297  8582 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
W20260709 13:04:29.015450 15635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.586s	user 0.007s	sys 0.007s
I20260709 13:04:29.035306  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.037190  8583 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:29.037492  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.016005 15742 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:29.046089  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.046815  8727 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:29.047892  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.049520  8725 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:29.050623 15742 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.051393 15700 consensus_queue.cc:1059] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:04:29.053563 15724 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.299s	user 0.000s	sys 0.001s
I20260709 13:04:29.053714  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:29.054734 15680 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:04:29.056385  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.048475  8727 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:29.057689  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.060444  8729 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:29.057720  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.062988  8581 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:29.034750 15662 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:29.020921 15596 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:04:29.076184  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:29.077039  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 46598.2us
I20260709 13:04:29.077952  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.086200  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:29.087980  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 350589us
I20260709 13:04:29.090054 15747 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:29.043254 15741 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:29.090799  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:29.092002  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.044721 15614 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:29.093338  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43648 (request call id 1516) took 2526 ms (client timeout 1922 ms). Trace:
W20260709 13:04:29.093765  9323 rpcz_store.cc:269] 0709 13:04:26.567218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.577319 (+ 10101us) service_pool.cc:224] Handling call
0709 13:04:29.093184 (+2515865us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b5400551c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:29.093284 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1337264512}
I20260709 13:04:29.053227 15700 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 13:04:29.092496  8726 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:04:29.081763 15739 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:04:29.101822  8726 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:29.080334 15662 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:29.095678 15736 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 13:04:29.039450 15678 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.117643  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:29.101806  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.085107 15596 consensus_queue.cc:1243] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:29.115636  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.112380  8726 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:29.111132 15732 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 13:04:29.029362 15727 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.123318  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.110340 15614 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.124390  9323 raft_consensus.cc:1275] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:29.125157  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.127374  8726 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:29.135779 15745 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 13:04:29.080931 15020 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:29.141872 15738 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 13:04:29.120369 15635 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.145177 15635 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.145767 15712 raft_consensus.cc:3055] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:29.146297 15712 raft_consensus.cc:740] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:29.147295 15712 consensus_queue.cc:260] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.148211 15635 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.104359 15661 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.148829 15712 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:29.131326  8729 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 18
W20260709 13:04:29.150458 15728 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.280s	user 0.000s	sys 0.001s
I20260709 13:04:29.117929 15478 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.128793  9331 raft_consensus.cc:1240] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:29.125485  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.104071 15504 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.156018  9330 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:29.123760  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.157882  9322 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:29.154477 15685 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.120541 15742 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:29.163393  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.173934  9322 raft_consensus.cc:2410] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:04:29.173182  8729 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:29.177677  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: true dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:29.178299  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.178509  9324 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:29.097062 15568 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.178932  8576 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:29.176617  8625 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:29.180341  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.159430 15614 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.192305  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:29.195308  9322 raft_consensus.cc:2376] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19. Re-sending same reply.
W20260709 13:04:29.195484  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1185) took 2529 ms (client timeout 2062 ms). Trace:
W20260709 13:04:29.195878  8659 rpcz_store.cc:269] 0709 13:04:26.666361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.232543 (+566182us) service_pool.cc:224] Handling call
0709 13:04:29.194200 (+1961657us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b5400420c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:29.195424 (+  1224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71830784}
I20260709 13:04:29.186487 15478 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:29.198920  8659 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:29.203487  8726 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:29.212932 15614 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.213656  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 3018) took 1970 ms. Trace:
I20260709 13:04:29.266841 15667 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:04:29.221155 15543 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.507s	user 0.007s	sys 0.000s
I20260709 13:04:29.224951  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.301349  8650 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:04:29.227010 15568 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.235297  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.236812  9324 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:29.244402  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.244546  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.250068  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.309737  8575 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:29.270437  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:29.283437  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.311791  8733 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.312323  8657 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:29.234761 15662 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.279680 15761 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.220631 15727 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.245076 15478 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:29.296298  8730 rpcz_store.cc:276] 0709 13:04:27.243356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.246392 (+  3036us) service_pool.cc:224] Handling call
0709 13:04:29.213452 (+1967060us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400415d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:29.213593 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2657463040}
I20260709 13:04:29.302639  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:29.303804  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.322628  8725 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:29.305241  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:29.306689  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.317087  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.298319 15667 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:29.335038  8725 pending_rounds.cc:85] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 12
I20260709 13:04:29.335474  8725 pending_rounds.cc:107] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 13:04:29.335793  8725 raft_consensus.cc:2889] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:29.336362  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:29.338300  9297 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:29.308316 15543 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:29.315578 15687 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.003s
W20260709 13:04:29.328487  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.301085 15614 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 13:04:29.350548  9329 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
W20260709 13:04:29.353917 15762 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:04:29.354861 15598 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:29.345623 15727 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:04:29.333904 15685 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.343112 15596 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:29.344074 15754 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:29.346634 15682 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.342293  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:29.371227  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:29.332851  8733 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
W20260709 13:04:29.360898  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 90743.3us
I20260709 13:04:29.344691 15737 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:29.364686  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:29.347694 15678 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:29.374686  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:29.380251  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:29.359759 15727 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:29.389861 15727 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:29.347175 15766 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.033s
W20260709 13:04:29.398000 15765 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
W20260709 13:04:29.398730 15767 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:04:29.373982 15737 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:29.334532 15761 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.347826 15687 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.405239  8652 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:29.405860 15568 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:29.407402 15687 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.374298 15413 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 13:04:29.409859 15761 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:29.410284 15413 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:29.410758 15413 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:29.411773 15413 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.384776 15515 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:29.405088 15737 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:29.416796 15690 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:29.417445 15690 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:29.417662 15413 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:29.429242  8582 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 13:04:29.432579 15759 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.234s	user 0.002s	sys 0.000s
I20260709 13:04:29.369988 15754 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:29.384542 15543 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:29.401903 15661 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:29.376708 15611 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.067s
I20260709 13:04:29.451006 15756 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:29.451802 15769 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:04:29.459518 15662 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:29.461826 15710 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.745s	user 0.010s	sys 0.002s
I20260709 13:04:29.462581  8575 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:29.438769 15478 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.383313 15667 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.466367 15667 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:29.469594 15504 consensus_queue.cc:1175] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1dc41db601fe4b91aa6d461b504da0e7" responder_term: 16 status { last_received { term: 16 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 16 index: 10 } }
I20260709 13:04:29.443081 15543 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:29.473809 15543 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:29.387423 15682 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.479792 15682 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.443449 15754 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.484629 15737 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.462330 15710 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:29.489244  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 110585us
I20260709 13:04:29.491700 15611 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:29.492836 15515 raft_consensus.cc:3055] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:29.493247 15515 raft_consensus.cc:740] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:29.446939 15661 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:29.494267 15661 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.494386 15515 consensus_queue.cc:260] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.489005 15568 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:29.501873 15568 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:29.504910 15720 consensus_meta.cc:287] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.685s	user 0.009s	sys 0.002s
I20260709 13:04:29.505453 15720 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.503162 15574 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.507112  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.507903  8655 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:29.508325  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3471) took 1017 ms. Trace:
I20260709 13:04:29.508733  8655 rpcz_store.cc:276] 0709 13:04:28.490702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.671779 (+181077us) service_pool.cc:224] Handling call
0709 13:04:29.507039 (+835260us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b5c00155b0c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140d580161 00007f140d721a79 00007f140ebdb9b6 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:29.508260 (+  1221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50726656}
I20260709 13:04:29.509502 15720 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:29.509630 15726 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.776s	user 0.003s	sys 0.000s
I20260709 13:04:29.502918 15568 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.514209  8550 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:29.518483  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 143916us
I20260709 13:04:29.519380  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:29.514912 15719 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.588s	user 0.006s	sys 0.006s
I20260709 13:04:29.521840  8727 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:29.519537 15611 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:29.524560 15611 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:29.524993 15611 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.525070  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.517506 15568 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:29.531426 15595 raft_consensus.cc:3055] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:29.531919 15595 raft_consensus.cc:740] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:29.534546  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.535154 15710 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.537010  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.537436 15464 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:29.539858  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.540002  8727 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:29.546661  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.508304 15727 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.548858  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.552554  8733 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:29.552999  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.555270 15574 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:29.555855 15574 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.532922 15595 consensus_queue.cc:260] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.558574 15574 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.559429 15776 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:29.559656 15681 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.560158 15776 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.562549 15776 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.564805 15665 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:29.521157 15720 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:29.565673 15720 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.567754 15720 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:29.569602 15727 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.570596  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.572154 15611 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.523626 15719 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:29.574520  8732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 3013) took 2449 ms (client timeout 2093 ms). Trace:
W20260709 13:04:29.574949  8732 rpcz_store.cc:269] 0709 13:04:27.125003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.125786 (+   783us) service_pool.cc:224] Handling call
0709 13:04:29.574296 (+2448510us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54004731d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:29.574446 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3214999808}
I20260709 13:04:29.575482 15719 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.579715  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:29.577291 15737 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.583598  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.582906  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.587356  8725 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:29.588122  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:29.591143  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:29.597688  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.600530  8727 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 15: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:29.600814  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.608243  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:29.610042  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.613390  8659 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 18. Ops: []
I20260709 13:04:29.574551 15611 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:29.614246  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.612536  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:29.594159 15727 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 13:04:29.594763 15574 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 13:04:29.595763 15667 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 13:04:29.574517 15754 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.724234  8655 raft_consensus.cc:2393] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:29.651799  8579 raft_consensus.cc:3055] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:29.651543  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.724947  8579 raft_consensus.cc:740] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:29.637454  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.615075  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:29.649551  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:29.595480 15569 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:29.651126  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.600943 15665 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:29.669683  8733 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:29.641731  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.640111  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.604410 15598 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:29.729600  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.738583  8550 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:29.741685  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.665369  8582 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:29.733461  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:29.733819  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 84400.2us
I20260709 13:04:29.743008  8583 raft_consensus.cc:2410] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 19 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 13:04:29.732841  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.744419  8655 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:29.738435  8579 consensus_queue.cc:260] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.736256  8576 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:29.745538  8579 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.739307  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.670562  8727 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:29.711463  8725 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:29.747344  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:29.733263  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 118323us
I20260709 13:04:29.745278  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:29.746279 15779 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 13:04:29.751418 15568 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:29.753756 15598 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:29.753648 15783 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:29.754580 15783 consensus_queue.cc:1243] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:29.755028 15568 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:29.755388 15568 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:29.756213 15568 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.757867 15611 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:29.758826 15681 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:29.759294  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:29.759984 15681 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:29.760185 15611 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:29.760394 15681 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:29.744029 15691 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 13:04:29.761389 15681 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.740607  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.738454  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.746688  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:29.766456  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.748503  9297 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:29.714223 15599 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:29.603780 15690 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.768618  8728 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 13:04:29.769145  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.732244 15777 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.609541 15776 consensus_queue.cc:1059] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.142s
I20260709 13:04:29.735984 15610 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:29.771797  8584 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:29.772884  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.771517  8728 pending_rounds.cc:85] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 8
I20260709 13:04:29.794308  8581 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:04:29.794631  8728 pending_rounds.cc:107] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 13:04:29.595163 15543 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.795125  8728 raft_consensus.cc:2889] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:29.773489  8583 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:29.795154  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:29.800143  8581 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:29.800889  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:29.652417 15661 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:29.775745  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.776525  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:29.802961  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.803211  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.810137  8658 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:29.764513 15568 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:29.811403  8658 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 13:04:29.812731  8658 raft_consensus.cc:1240] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:29.802423  8656 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 11: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:29.814255 15780 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:04:29.815393  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3472) took 1324 ms. Trace:
I20260709 13:04:29.815793  8657 rpcz_store.cc:276] 0709 13:04:28.490989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.968272 (+477283us) service_pool.cc:224] Handling call
0709 13:04:28.995783 (+ 27511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.314750 (+318967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.815161 (+500411us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54000c85d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:29.815339 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":727463296}
I20260709 13:04:29.774556 15777 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.821822  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.823772  8581 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:29.773733 15362 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:29.845943  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.846874  8576 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:29.848027  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:29.848484  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.776705 15681 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:29.861768  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:29.858326  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:29.865043  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 88620.5us
I20260709 13:04:29.865861  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:29.774206 15599 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:29.859210  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:29.827651 15362 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:29.869140  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:29.818050 15601 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 13:04:29.764068 15598 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.874933  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.888927  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.882158  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3066) took 1337 ms. Trace:
I20260709 13:04:29.893328  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:29.894570  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:29.896091  8576 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:29.896333  8728 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:29.897364  8583 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:29.897790  8659 rpcz_store.cc:276] 0709 13:04:28.544735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.210663 (+665928us) service_pool.cc:224] Handling call
0709 13:04:29.210970 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.613815 (+402845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.881929 (+268114us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b5400001754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:29.882097 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":924995840}
I20260709 13:04:29.900283  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:29.867436 15694 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:29.903139  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:29.880019 15782 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 13:04:29.870754 15664 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:29.905718 15776 consensus_queue.cc:1059] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:04:29.911657  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.878453 15690 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.912436  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.912681 15362 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:29.874256 15432 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:29.873546 15667 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:29.916411  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:29.918579  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:29.919198  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:29.910745 15709 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:29.879493 15754 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.924957  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:29.942340 15784 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 13:04:29.942044 15700 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.954283  8583 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 22.
W20260709 13:04:29.971140  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.981009 15661 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:29.981485 15504 raft_consensus.cc:3055] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:29.945899 15640 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:04:29.989758  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:29.887015 15712 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:29.995324  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 76204.5us
I20260709 13:04:29.997150  9327 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:30.003598  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:30.019793  9325 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.930042 15623 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:29.930204 15511 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:30.006283  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 81414.8us
I20260709 13:04:29.906646 15685 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.088s
I20260709 13:04:30.017120  9330 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:30.015120  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:30.016309  8576 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:30.022720  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:30.028961  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:30.031461  8580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3346) took 1019 ms. Trace:
I20260709 13:04:30.005453 15664 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:30.005959 15710 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:29.993600 15643 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 13:04:30.032040  8580 rpcz_store.cc:276] 0709 13:04:29.012334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.013030 (+   696us) service_pool.cc:224] Handling call
0709 13:04:29.013278 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.166151 (+152873us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.166182 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.166229 (+    47us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:29.166269 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.031158 (+864889us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54001685d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:30.031211 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.031382 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":351140736}
I20260709 13:04:30.013970 15504 raft_consensus.cc:740] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:30.008548 15610 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:04:30.021507 15623 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:30.057893  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:30.031073 15661 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:30.014216 15700 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.047479 15605 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:30.081794  8623 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:30.041944 15432 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.046154 14906 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:30.093295  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70547us
I20260709 13:04:30.094128 15792 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.056918 15623 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.048367 15643 consensus_queue.cc:1243] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:04:30.095920  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:30.098388  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:30.116993  8622 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:30.050052 15710 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:30.094331 15785 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 13:04:30.057056 15720 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 13:04:30.034722 15511 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.122898  8659 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:04:30.086598 15605 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.126998  9327 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:30.127085  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.088260 15661 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.143020  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:30.112133  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.145794  8659 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:30.147820  9327 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:30.134943  8579 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 13:04:30.136531  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.069185  9330 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:30.149507  8579 pending_rounds.cc:85] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Aborting all ops after (but not including) 9
I20260709 13:04:30.115833 14906 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.149807  8579 pending_rounds.cc:107] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 13:04:30.155120  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:30.157294  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45285.6us
I20260709 13:04:30.132534 15729 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:30.161989  9331 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:04:30.163497  8652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1179) took 3503 ms (client timeout 2490 ms). Trace:
I20260709 13:04:30.163326 15729 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:30.163918  8652 rpcz_store.cc:269] 0709 13:04:26.659662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.555814 (+896152us) service_pool.cc:224] Handling call
0709 13:04:30.163342 (+2607528us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54003223d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:30.163436 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4241723776}
I20260709 13:04:30.144129 15605 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:30.174769  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:30.161839  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.165107  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.169992  8579 raft_consensus.cc:2889] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:30.115545 15432 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.171774  8731 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
W20260709 13:04:30.173058  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43760.2us
I20260709 13:04:30.165439 15729 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:30.155016 15640 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:30.160166 14906 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:30.153756 15793 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:30.123759 15792 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.201249  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:30.201912  9331 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:30.159740 15599 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:30.197069  8659 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:30.197561  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 3045) took 1015 ms. Trace:
I20260709 13:04:30.161796 15664 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:30.206857  8729 rpcz_store.cc:276] 0709 13:04:29.181959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.182292 (+   333us) service_pool.cc:224] Handling call
0709 13:04:29.183528 (+  1236us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.184003 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.184035 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.184585 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.817099 (+632514us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b180017d940. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:29.862052 (+ 44953us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:29.862088 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.862143 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.187799 (+325656us) raft_consensus.cc:1685] finished
0709 13:04:30.187820 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.197474 (+  9654us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":271,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4170,"spinlock_wait_cycles":8614272,"thread_start_us":44780,"threads_started":1,"wal-append.queue_time_us":712843}
I20260709 13:04:30.186064 15400 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:04:30.218272 15664 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.228104  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:30.229951  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:30.232574  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.234588  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:30.235412  9323 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:30.237561  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:30.238570  9322 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:30.180754 15685 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 13:04:30.242908  9327 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:30.253929  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:30.226477 15758 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.150s	user 0.006s	sys 0.000s
W20260709 13:04:30.257431  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 175699us
I20260709 13:04:30.206311 15643 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.271369  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:30.283187  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:30.225361 15664 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.287813  9325 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:30.294205 15792 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:30.295382 15794 log.cc:927] Time spent T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 13:04:30.296756 15790 log.cc:927] Time spent T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260709 13:04:30.296882 15760 log.cc:927] Time spent T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.048s	user 0.006s	sys 0.000s
I20260709 13:04:30.187389 15694 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:04:30.242082 15599 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:30.303773  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:30.155293 15504 consensus_queue.cc:260] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.306037  8578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41992 (request call id 2824) took 1228 ms. Trace:
I20260709 13:04:30.306313 15504 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:30.306607  8578 rpcz_store.cc:276] 0709 13:04:29.077986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.078854 (+   868us) service_pool.cc:224] Handling call
0709 13:04:29.079153 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.079544 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.079576 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.079948 (+   372us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.123842 (+ 43894us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:29.123881 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.123955 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.241967 (+1118012us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:30.305636 (+ 63669us) raft_consensus.cc:1685] finished
0709 13:04:30.305669 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.305948 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":78,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"spinlock_wait_cycles":43076864,"thread_start_us":42853,"threads_started":1,"wal-append.queue_time_us":109177}
I20260709 13:04:30.311421 15543 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.305261  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:30.320690 15691 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:30.320912  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:30.321885 15792 consensus_queue.cc:1048] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:04:30.328753  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 45706.8us
I20260709 13:04:30.329439  9331 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 13.
W20260709 13:04:30.255586 15791 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 13:04:30.329697  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:30.332739  8623 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:30.330116  9327 raft_consensus.cc:1240] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
W20260709 13:04:30.337916  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 171903us
I20260709 13:04:30.348425  8623 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:30.355494  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.361851  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:30.291535  9323 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:30.346158 15776 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.268513 15413 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.117s
W20260709 13:04:30.401789  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 177983us
I20260709 13:04:30.353205 15792 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:30.284698 15720 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.112s
I20260709 13:04:30.353588 15569 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.328542 15568 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
W20260709 13:04:30.407186  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 175926us
I20260709 13:04:30.327160 15643 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.387742 15691 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:30.411749  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 50016us
I20260709 13:04:30.415916  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:30.412070  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:30.399718 15543 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:30.404569 15640 raft_consensus.cc:697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:30.441921  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.406067 15792 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:30.414381 15601 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:30.454998  8623 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:30.437727 14906 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.451498  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.461653  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.415551 15710 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.469961  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:30.468040  8733 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:30.470908  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:30.473213  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:30.466248  8652 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:30.433866 15720 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:30.435472 15705 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.481150 14906 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:30.476158  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.488838  8549 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:30.465058 15792 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:30.494601  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 138815us
I20260709 13:04:30.428014 15709 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
W20260709 13:04:30.498634  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.495118  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.513677  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.435621 15596 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
W20260709 13:04:30.490185 15598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.006s	sys 0.019s
I20260709 13:04:30.436085 15543 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.479998 15413 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.485479 15601 raft_consensus.cc:697] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:30.479311 15661 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.482116 15568 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:30.496951 15666 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:30.520984 15598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.006s	sys 0.019s
W20260709 13:04:30.533344  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 67034.3us
I20260709 13:04:30.490309 15729 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 13:04:30.534283 15810 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:30.534631 15640 consensus_queue.cc:237] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.515630 15432 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.526228 15568 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:30.560676 15605 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.573848  8725 raft_consensus.cc:1240] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 13:04:30.576555 15605 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:30.565368  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70340.2us
I20260709 13:04:30.547112 15511 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.558578 15803 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:04:30.559015 15710 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:30.565060 15413 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:30.566179 15815 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:30.595496 15815 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:30.595881 15815 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.596264 15710 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:30.596657 15601 consensus_queue.cc:237] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.597729 15826 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.598207 15826 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.567909 15667 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:04:30.572185 15568 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:30.616770  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.619153 15623 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.619827 15643 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.608090 15413 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:30.621585  8729 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:30.621999 15712 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:30.622493 15712 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:30.587742 15432 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:30.622587 15413 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.623616 15432 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.592757 15538 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.639698 15538 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.640426 15719 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:30.640862 15719 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:30.642194 15538 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.642803 15515 raft_consensus.cc:3060] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:30.643077 15400 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:30.643431 15400 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:30.641842 15719 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:30.651058  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.660584  8728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:30.661214  8728 raft_consensus.cc:606] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.652268 15826 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.651930 15792 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.652398 15432 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.662936  8582 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:30.669222  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3062) took 2542 ms. Trace:
W20260709 13:04:30.664484  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 198385us
I20260709 13:04:30.670424  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:30.671865  8582 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 20
W20260709 13:04:30.675892  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 220934us
I20260709 13:04:30.677016  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:30.608415 15807 consensus_queue.cc:1059] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
I20260709 13:04:30.683398  8654 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:30.684334  8582 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:30.687011  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:30.688673  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:30.687757  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 264961us
I20260709 13:04:30.690310  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.691118  8650 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:04:30.691663  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.692173  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.693547  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.695472  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:30.695590  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:30.697367  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.696305  8582 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.698341  8696 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:30.682092  8579 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 21. Current term is 22. Ops: [19.11-21.12]
I20260709 13:04:30.689579  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:30.668152 15623 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:30.697510  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.672147 15808 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:30.676477 15719 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:30.670092  8651 rpcz_store.cc:276] 0709 13:04:28.126927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.767682 (+640755us) service_pool.cc:224] Handling call
0709 13:04:28.768048 (+   366us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:28.781044 (+ 12996us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.781098 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:28.792757 (+ 11659us) log.cc:844] Serialized 52 byte log entry
0709 13:04:28.859689 (+ 66932us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:28.942294 (+ 82605us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.470824 (+1528530us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b18004166e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406aa3c0a 00007f14101e3219 00007f1406b2ba40 00007f1406b264e4 00007f1406b26455 00007f140d60e3c7 00007f1409c209a7 00007f1409c300bf 00007f1409c30038
0709 13:04:30.471074 (+   250us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.668335 (+197261us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.668566 (+   231us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.668776 (+   210us) raft_consensus.cc:1685] finished
0709 13:04:30.668819 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.669100 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1583229,"prepare.queue_time_us":1625415,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":95,"raft.queue_time_us":21508,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":9468,"spinlock_wait_cycles":3808000,"thread_start_us":206155,"threads_started":2,"wal-append.queue_time_us":125529,"wal-append.run_cpu_time_us":3489,"wal-append.run_wall_time_us":1204386}
I20260709 13:04:30.721154  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:30.720337 15643 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:30.666224 15400 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.726135 15623 consensus_queue.cc:1059] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:30.727311 15623 raft_consensus.cc:3055] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:30.727779 15623 raft_consensus.cc:740] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:30.667356 15712 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.728915 15623 consensus_queue.cc:260] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.730970 15829 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.731599 15623 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:30.731364 15808 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.732537  8652 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:30.714145 15814 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:30.734418 15814 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.663872 15809 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260709 13:04:30.738257 15664 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.740480 15664 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.726567 15643 raft_consensus.cc:697] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:30.741619 15543 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.686178 15687 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:30.741932 15643 consensus_queue.cc:237] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:30.744160 15789 log.cc:927] Time spent T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.583s	user 0.000s	sys 0.001s
I20260709 13:04:30.746990  8655 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:30.730400 15792 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:30.745611  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48265.2us
I20260709 13:04:30.750517  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "a0d703ee18464475a290425976246555"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:30.756619  8731 raft_consensus.cc:606] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Received request to transfer leadership to TS a0d703ee18464475a290425976246555
I20260709 13:04:30.753299  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:30.754839  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 274853us
I20260709 13:04:30.757000  9296 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:30.685972 15538 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:30.770030 15538 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:30.771621  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 207092us
I20260709 13:04:30.772787 15538 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.776888  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:30.777652  8726 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
W20260709 13:04:30.777676  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 88164.8us
I20260709 13:04:30.713264 15598 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.792589  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:30.763796  8582 raft_consensus.cc:2468] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:30.757045  8731 raft_consensus.cc:614] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:04:30.782272  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:30.726225 15811 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.133s	user 0.001s	sys 0.001s
I20260709 13:04:30.796967  8696 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:30.784538  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:30.798748  8655 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:30.737246 15814 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:30.731595 15829 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.793622  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:30.802897  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:30.809561  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52623.8us
I20260709 13:04:30.809494  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:30.811554  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.812203  9331 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:30.814210  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.713551 15709 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.087s
I20260709 13:04:30.803609  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:30.804000  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.815560  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:30.817509  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52183.8us
I20260709 13:04:30.820056  8652 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:30.817317  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.821918  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.754135 15823 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.826625  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.827260  8658 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:30.827543  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3060) took 2701 ms. Trace:
I20260709 13:04:30.828238  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.713716 15803 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.095s
I20260709 13:04:30.798318 15303 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:30.833328  8731 raft_consensus.cc:1240] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:30.834424  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:30.836043  9325 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.790921 15820 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.813651 15816 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.825300 15825 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:30.804800 15599 raft_consensus.cc:2804] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:30.852347 15830 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:30.853011 15830 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:30.854617 15796 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.501s	user 0.003s	sys 0.000s
I20260709 13:04:30.862373  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:30.864214  8581 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:30.865864 15816 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.867884  9325 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:30.827962  8653 rpcz_store.cc:276] 0709 13:04:28.126363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.760372 (+634009us) service_pool.cc:224] Handling call
0709 13:04:28.777034 (+ 16662us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:28.787040 (+ 10006us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.787092 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:28.787795 (+   703us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.009744 (+221949us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:29.497496 (+487752us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b48001b2358. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140eb3faa1 00007f140eb36b37 00007f140eb369ca 00007f140ec2e26f 00007f140ec2df37 00007f140ec4255c 00007f140ec42508 00007f140ec42488 00007f140ec42443 00007f140ec416ec 00007f140db6ba96
0709 13:04:30.676406 (+1178910us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.819037 (+142631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.819160 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.819223 (+    63us) raft_consensus.cc:1685] finished
0709 13:04:30.827356 (+  8133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.827480 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1491440,"prepare.queue_time_us":1557086,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":15529,"raft.run_cpu_time_us":2418,"raft.run_wall_time_us":711460,"spinlock_wait_cycles":13931776,"thread_start_us":474708,"threads_started":3,"wal-append.queue_time_us":221846,"wal-append.run_cpu_time_us":2672,"wal-append.run_wall_time_us":1098009}
I20260709 13:04:30.843344  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:30.830708 15504 consensus_meta.cc:287] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.524s	user 0.011s	sys 0.002s
I20260709 13:04:30.835310 15826 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:30.876891  8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3080) took 1013 ms. Trace:
I20260709 13:04:30.839407 15666 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:30.878057  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:30.834789 15800 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.877146  8579 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:30.884666  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.890795  8731 raft_consensus.cc:606] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.867602 15825 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:30.870442 15605 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.882296  8656 rpcz_store.cc:276] 0709 13:04:29.863387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.863551 (+   164us) service_pool.cc:224] Handling call
0709 13:04:29.863845 (+   294us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:29.864192 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.864222 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:29.871718 (+  7496us) log.cc:844] Serialized 52 byte log entry
0709 13:04:29.992859 (+121141us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:30.828624 (+835765us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.869131 (+ 40507us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.876556 (+  7425us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.876648 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.876692 (+    44us) raft_consensus.cc:1685] finished
0709 13:04:30.876705 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.876840 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":565480,"prepare.queue_time_us":681098,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":59277,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":2839,"thread_start_us":362522,"threads_started":2,"wal-append.queue_time_us":240193}
I20260709 13:04:30.848347 15832 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:30.916597  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:30.871912 15830 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.883633 15826 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.880739 15793 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.928613  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:30.929261  8656 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.884874 15599 raft_consensus.cc:697] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:30.927075  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:30.924487  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 101393us
I20260709 13:04:30.948752  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
I20260709 13:04:30.949522  8731 raft_consensus.cc:606] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.954877  9324 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:30.919319 15825 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:30.884373 15820 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 13:04:30.932320 15826 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:30.930294 15792 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.962071  8656 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:30.915995 15666 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.963425  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.964108  8656 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:30.965342  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:30.926709 15809 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
W20260709 13:04:30.957429  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 80048.3us
I20260709 13:04:30.957672  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:30.927863 15605 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.926527 15832 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.972488 15599 consensus_queue.cc:237] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:30.973196 15838 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 13:04:30.973594 15666 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:30.976652 15816 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.977356  8731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54"
dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
mode: GRACEFUL
new_leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:43176
W20260709 13:04:30.977931 15843 log.cc:927] Time spent T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:04:30.981665 15605 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:30.970212 15504 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:30.982403 15504 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:30.982828 15504 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:30.971596 15815 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.985996  8575 raft_consensus.cc:1275] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:30.986469  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:30.987169  8655 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:30.987833 15815 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:30.988683  8577 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:30.988704  8575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3371) took 1338 ms. Trace:
I20260709 13:04:30.991969  8575 rpcz_store.cc:276] 0709 13:04:29.650344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.771335 (+120991us) service_pool.cc:224] Handling call
0709 13:04:29.771657 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.986461 (+1214804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.988466 (+  2005us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540030be54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:30.988643 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2791775616}
I20260709 13:04:30.993764  8651 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 22.
I20260709 13:04:30.996446 15829 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:30.997367 15841 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:04:30.997788  8655 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:31.004189 15568 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:30.964291 15792 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:30.998952 15832 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.005915  8655 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
W20260709 13:04:31.024523  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.028517  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.036782  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:30.965150 15362 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:04:31.034459  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.030555  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.059155  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.060039  8655 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:31.045591  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.064661  8658 raft_consensus.cc:1240] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:31.038856 15809 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:31.069922  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.064586  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:31.068219  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 177828us
I20260709 13:04:31.069636  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.058033 15720 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:31.061066 15362 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:31.063638 15832 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:31.042032 15605 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 13:04:31.065119 15709 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:31.073290 15815 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:31.082477  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.078662 15851 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:31.086047  8698 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.076951 15832 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:31.084131  8655 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:31.081378 15815 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:31.089977 15815 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.075911 15826 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.092458  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:31.093310  8653 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:31.095013 15792 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.095161  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:31.095952  8726 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:31.080602 15515 consensus_queue.cc:1059] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 13:04:31.099078  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.102088 15832 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:31.098199 15825 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.092547 15826 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.109364 15568 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.110205 15362 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.100870 15504 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.111523 15362 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:31.116276  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.119735  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.119104 15851 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:04:31.122759 15504 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.124418 15777 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.081230 15679 consensus_queue.cc:1048] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 13:04:31.104678 15825 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.129307  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3175) took 1398 ms. Trace:
I20260709 13:04:31.132877  8726 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:31.133638  8651 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:31.134769  8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1199) took 1001 ms. Trace:
I20260709 13:04:31.135361  8659 rpcz_store.cc:276] 0709 13:04:30.133245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.142704 (+  9459us) service_pool.cc:224] Handling call
0709 13:04:31.133220 (+990516us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b540025bf54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:31.133360 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4508032}
I20260709 13:04:31.137020  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.137578  9296 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:31.137737  8625 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.139600  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.140409  8659 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:31.145815  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.183907  8659 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:31.185582 15538 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.185530  8549 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.186408 15538 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.186872 15830 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:31.187927 15830 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:31.189143 15830 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:31.191927  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54450.3us
I20260709 13:04:31.192801 15820 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:31.193454 15820 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:31.194432 15820 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.209209 15574 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:31.209805 15574 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:31.210813 15574 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.212054 15574 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:31.215463  8658 pending_rounds.cc:85] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 14
I20260709 13:04:31.146672 15815 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.215950  8658 pending_rounds.cc:107] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 13:04:31.216328  8658 raft_consensus.cc:2889] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:31.218135 15815 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.128517 15809 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:31.248934  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.130070 15829 raft_consensus.cc:3055] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:31.169661 15640 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:31.185767 15737 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.294463 15856 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.292994 15851 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:31.141861 15643 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.165s
I20260709 13:04:31.300246 15829 raft_consensus.cc:740] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:31.311403 15719 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:31.336299 15825 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.312952 15737 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.327260 15815 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.313365 15640 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:31.369541 15719 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.379076  8726 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 13:04:31.379505  8726 pending_rounds.cc:85] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 11
I20260709 13:04:31.379750  8726 pending_rounds.cc:107] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 13:04:31.379997  8726 raft_consensus.cc:2889] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:31.381915  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.329205 15851 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.189018 15538 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:31.383692  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.370774 15777 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.371122 15850 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:31.391096  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.392243  8659 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.392012 15850 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.393061 15824 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:31.394289 15850 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.372450 15815 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.395341 15777 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.397562 15777 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.385782 15832 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.400132 15832 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.401494 15815 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:31.389945 15859 log.cc:927] Time spent T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:04:31.371526 15825 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.403697 15851 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.404575 15825 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.387162 15829 consensus_queue.cc:260] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.406203 15829 raft_consensus.cc:3060] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:31.369033 15737 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:31.402019 15719 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.326709 15856 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.428133 15856 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:31.415813 15799 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.941s	user 0.003s	sys 0.006s
I20260709 13:04:31.374670 15640 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.417721  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.448473  8652 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:31.424140 15569 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:31.453387  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:31.455418  8659 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:31.456815  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.457582  8652 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:31.457651  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:31.458312  9330 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:31.447860 15574 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:31.460068  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.450595 15820 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.466653 15820 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.471918  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3374) took 1598 ms. Trace:
I20260709 13:04:31.472347  8576 rpcz_store.cc:276] 0709 13:04:29.873566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.058810 (+185244us) service_pool.cc:224] Handling call
0709 13:04:30.088294 (+ 29484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:30.088558 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:30.088581 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:30.095567 (+  6986us) log.cc:844] Serialized 26 byte log entry
0709 13:04:30.267087 (+171520us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:30.267133 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.267202 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:31.415055 (+1147853us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:31.471618 (+ 56563us) raft_consensus.cc:1685] finished
0709 13:04:31.471640 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.471859 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35003776,"thread_start_us":128856,"threads_started":1,"wal-append.queue_time_us":273509}
I20260709 13:04:31.469058 15820 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:31.481487  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.460140  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:31.460186  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.461285  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:31.489974  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.490760  8659 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:31.450441 15569 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:31.496429  8652 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:31.502951  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:31.506079 15786 log.cc:927] Time spent T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.388s	user 0.009s	sys 0.000s
I20260709 13:04:31.507973  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3079) took 1650 ms. Trace:
I20260709 13:04:31.508386  8657 rpcz_store.cc:276] 0709 13:04:29.857906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.858033 (+   127us) service_pool.cc:224] Handling call
0709 13:04:29.858286 (+   253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.858630 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.858659 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.863381 (+  4722us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.995482 (+132101us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:30.589930 (+594448us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.590904 (+   974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.591030 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:31.505448 (+914418us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:31.507625 (+  2177us) raft_consensus.cc:1685] finished
0709 13:04:31.507733 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.507909 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":541237,"prepare.queue_time_us":657148,"raft.queue_time_us":18233,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":27515,"spinlock_wait_cycles":37576704,"thread_start_us":196490,"threads_started":2,"wal-append.queue_time_us":197375}
W20260709 13:04:31.511152 15840 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.649s	user 0.002s	sys 0.008s
I20260709 13:04:31.515281 15709 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:31.515092 15569 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.515946 15709 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.517876  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.519109 15667 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:31.519531 15667 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:31.519856 15667 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:31.521073  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:31.484301 15413 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:31.522751  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:31.523689  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:31.524695  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.531255  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.531894  8728 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:31.529009  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.497236 15829 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:31.537285  8724 raft_consensus.cc:2393] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:31.532958  8726 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:31.496050 15800 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:31.538872  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.546275  9329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1294) took 1411 ms. Trace:
I20260709 13:04:31.548755  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3474) took 3057 ms. Trace:
W20260709 13:04:31.549049 15842 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.482s	user 0.004s	sys 0.004s
I20260709 13:04:31.513245 15720 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.557641 15413 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:31.559916 15709 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.567809  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.570070  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:31.560106 15700 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.489781 15574 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.562469  9329 rpcz_store.cc:276] 0709 13:04:30.134582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.134727 (+   145us) service_pool.cc:224] Handling call
0709 13:04:30.135012 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.161109 (+ 26097us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:30.184730 (+ 23621us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.654937 (+470207us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.656999 (+  2062us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:30.658269 (+  1270us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:30.658694 (+   425us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.546034 (+887340us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b54008f0cd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:31.546082 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.546211 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":566,"prepare.queue_time_us":24557,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":60688,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":10376,"spinlock_wait_cycles":59411712,"thread_start_us":464495,"threads_started":2,"wal-append.queue_time_us":454638}
W20260709 13:04:31.572981  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 106464us
I20260709 13:04:31.573006  8659 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:31.573992  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.565124  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.574658  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.562515  8654 rpcz_store.cc:276] 0709 13:04:28.491576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.987144 (+495568us) service_pool.cc:224] Handling call
0709 13:04:28.987544 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.684916 (+1697372us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.548433 (+863517us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400380254. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:31.548657 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3899345792}
I20260709 13:04:31.574692  8656 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
W20260709 13:04:31.567076  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.562860  8727 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 22: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:31.575882  8654 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:31.571089  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.580381  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.565436 15829 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:31.590109  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.590816  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.597369  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.597911  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.601895  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.594357  8726 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
I20260709 13:04:31.595125 15800 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:31.598218  8659 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 22: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:31.600389  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.577972 15777 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.629568 15777 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:31.630319  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.631788 15777 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.537266 15776 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 13:04:31.633515 15776 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:31.633905 15776 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:31.634773 15776 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.636076 15776 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:31.639070  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1534) took 1370 ms. Trace:
I20260709 13:04:31.639565  9324 rpcz_store.cc:276] 0709 13:04:30.268881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.269031 (+   150us) service_pool.cc:224] Handling call
0709 13:04:30.301888 (+ 32857us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.955516 (+653628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.638806 (+683290us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400541d54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:31.639002 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1501177088}
I20260709 13:04:31.639983 15829 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.566294 15720 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.647055 15720 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:31.626149 15413 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.628516 15574 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:31.649290  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:31.649534  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.593798  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.662982  8658 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:31.704077  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.705592  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:31.707176  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.707999  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.710134  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.726542  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:31.728142  8726 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
W20260709 13:04:31.735491  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.735863  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.739987  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:31.744215 15853 log.cc:927] Time spent T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.678s	user 0.002s	sys 0.007s
I20260709 13:04:31.748555 15712 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:31.727181 15856 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.751050 15856 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.752684  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1532) took 1492 ms. Trace:
I20260709 13:04:31.732339 15829 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.753085  9327 rpcz_store.cc:276] 0709 13:04:30.259862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.260192 (+   330us) service_pool.cc:224] Handling call
0709 13:04:30.329939 (+ 69747us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.349659 (+ 19720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.752450 (+1402791us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b5400a06250. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:31.752626 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159571200}
I20260709 13:04:31.753899 15829 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.756418 15829 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.725281 15777 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.761358  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:31.761358  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.762015  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.767720  8726 raft_consensus.cc:2410] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:31.768153  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.765118  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.730633 15800 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:31.779412 15800 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.737377 15504 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.781955 15504 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.783742 15777 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.729794 15574 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.785388 15574 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.786275 15777 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.787273  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.787889 15574 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.790793  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.793049  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.766346 15802 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:31.794014  8727 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:31.794327 15802 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.794790  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.795301  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.795648  8728 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:31.729559 15863 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 13:04:31.796837  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.735951 15815 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.801581  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:31.802348  9327 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:31.798446  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.798470  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:31.796002  8727 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
W20260709 13:04:31.813555  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.762977 15515 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:31.816582  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.738466 15754 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.067s
I20260709 13:04:31.822979  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.824783  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.796789 15802 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.799414 15504 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:31.828358  8697 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:31.835127 15605 raft_consensus.cc:2764] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:04:31.829746  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.836997  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.837692 15777 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:31.838801 15777 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:31.839511  8579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34698 (request call id 1380) took 1057 ms. Trace:
I20260709 13:04:31.840180  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.840096  8579 rpcz_store.cc:276] 0709 13:04:30.781901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.784250 (+  2349us) service_pool.cc:224] Handling call
0709 13:04:31.839271 (+1055021us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b54001694d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:31.839416 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177560960}
I20260709 13:04:31.842025 15777 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:31.842510  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.843665  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 3062) took 1388 ms. Trace:
I20260709 13:04:31.843727  8698 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.844281  8725 rpcz_store.cc:276] 0709 13:04:30.455285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.455445 (+   160us) service_pool.cc:224] Handling call
0709 13:04:30.455754 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:30.574295 (+118541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.843300 (+1269005us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54004731d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:31.843580 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":270954752}
I20260709 13:04:31.845495  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:31.828831  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.849466  8652 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:31.858889  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.812940 15856 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.860996  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:31.822440 15574 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:31.871021  8623 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:31.830658 15825 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:31.830158 15756 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:31.880103  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:31.874406 15666 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20260709 13:04:31.860086 15605 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:31.812655 15815 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.863375 15863 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:31.864282 15596 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:31.875479 15750 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:31.860354 15802 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:31.865630 15636 consensus_queue.cc:1059] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 13:04:31.855129 15667 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:31.884994 15851 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:31.884049 15710 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:31.892999 15800 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:31.907337 15710 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:31.908886 15719 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:31.910009 15803 raft_consensus.cc:3055] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:31.910475 15803 raft_consensus.cc:740] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:31.908694 15710 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.914626  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.916808  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.917495  8657 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: [17.12-19.14]
I20260709 13:04:31.922470  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:31.924110  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.924166  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:31.925675  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 51575.8us
I20260709 13:04:31.928416 15884 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:31.924906  8654 raft_consensus.cc:2376] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 22. Re-sending same reply.
I20260709 13:04:31.930907 15574 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.934182  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:31.934708  8656 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:31.934813  8654 raft_consensus.cc:2376] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15. Re-sending same reply.
I20260709 13:04:31.934664 15574 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:31.906010 15400 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:31.923625 15851 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:31.947836  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 84939us
I20260709 13:04:31.931846 15478 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:31.951198  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:32.048647  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 74071.2us
I20260709 13:04:31.952157 15538 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:31.945794 15863 consensus_queue.cc:1059] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.102s
I20260709 13:04:32.063076  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:32.064764 15538 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.066128 15595 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:32.066854 15851 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.070185 15884 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:32.071652 15880 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:32.072031 15880 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:32.074206 15574 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:32.077423  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:32.048591 15596 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:32.086233  8698 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:32.083904 15661 raft_consensus.cc:3055] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:04:32.091398  8698 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:32.091544 15661 raft_consensus.cc:740] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:31.946892 15803 consensus_queue.cc:260] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.085915 15538 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:32.072871 15880 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:32.095417  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 215236us
I20260709 13:04:32.084441 15643 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:32.105027  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.092773 15596 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:32.108719 15875 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:04:32.108642 15661 consensus_queue.cc:260] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.106413 15803 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:32.130872  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:32.131151  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:32.133926  9327 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:32.110033 15880 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:32.137842  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46513.5us
I20260709 13:04:32.161736 15818 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 13:04:32.166324 15750 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:32.166931 15750 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:32.167912 15750 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.134902 15667 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.139344 15661 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:04:32.173882  8650 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:32.175925 15839 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.176429 15839 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.181249  8655 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:04:32.214247  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.174619 15643 raft_consensus.cc:2804] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:32.187410 15640 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:32.238168  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 107399us
I20260709 13:04:32.240409  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:32.241070  8725 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
W20260709 13:04:32.234378 15876 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 13:04:32.231912 15851 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:32.244041  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:32.244714  8726 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:32.234628 15643 raft_consensus.cc:697] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:32.253108  8550 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:32.248179 15885 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 13:04:32.256029  8729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 3061) took 1802 ms. Trace:
I20260709 13:04:32.233560 15640 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.259191  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:32.258842  8729 rpcz_store.cc:276] 0709 13:04:30.453942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.454145 (+   203us) service_pool.cc:224] Handling call
0709 13:04:30.463591 (+  9446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.621982 (+158391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.255756 (+1633774us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400338954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:32.255947 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":362878208}
W20260709 13:04:32.260577  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.222692 15596 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.246775 15851 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:32.266021  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:32.261123 15504 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:32.269937  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:32.268947 15667 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:32.274473 15886 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:32.275110 15886 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:32.275602 15886 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:32.277652 15856 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:32.278523 15768 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Append to log took a long time: real 1.882s	user 0.004s	sys 0.008s
I20260709 13:04:32.281715 15515 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:32.282502 15515 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:32.286720  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.288709  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3497) took 1521 ms. Trace:
I20260709 13:04:32.296712 15820 raft_consensus.cc:3055] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:32.303594 15808 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.306831 15729 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:32.307461 15729 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:32.308005  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:32.312057 15825 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:32.289361  8651 rpcz_store.cc:276] 0709 13:04:30.766923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.006690 (+239767us) service_pool.cc:224] Handling call
0709 13:04:31.133436 (+126746us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.134032 (+   596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.288442 (+1154410us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b540025bf50. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:32.288622 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":290688384}
I20260709 13:04:32.319274 15839 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.320971  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3091) took 1070 ms. Trace:
I20260709 13:04:32.326880  8655 rpcz_store.cc:276] 0709 13:04:31.250809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.388936 (+138127us) service_pool.cc:224] Handling call
0709 13:04:31.389207 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:32.181906 (+792699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.320632 (+138726us) spinlock_profiling.cc:243] Waited 791 ms on lock 0x7b5400352ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:32.320885 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1820363008}
I20260709 13:04:32.328347 15820 raft_consensus.cc:740] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:32.324306  8653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3505) took 1473 ms (client timeout 1657 ms). Trace:
W20260709 13:04:32.329720  8653 rpcz_store.cc:269] 0709 13:04:30.851212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.883064 (+ 31852us) service_pool.cc:224] Handling call
0709 13:04:32.324067 (+1441003us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54004704d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:32.324238 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":464188288}
I20260709 13:04:32.324417 15776 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:32.325065 15805 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 1.842s	user 0.004s	sys 0.021s
I20260709 13:04:32.269646 15504 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:32.377799 15504 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:32.308625 15729 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:32.384460 15899 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 13:04:32.337441  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:32.348071  9325 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:32.268250 15851 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:32.372483 15776 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:32.411270  8724 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.012s	sys 0.032s
I20260709 13:04:32.394877 15729 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:32.382620 15643 consensus_queue.cc:237] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.401728 14906 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.403280  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:32.416072 14906 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.416333  8730 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:32.406061  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:32.407378  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:32.410418  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.332064  8651 raft_consensus.cc:1275] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:04:32.412559  8724 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.012s	sys 0.032s
W20260709 13:04:32.414273  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 141246us
W20260709 13:04:32.414259 15599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.439s	user 0.017s	sys 0.009s
I20260709 13:04:32.428030  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:32.413623 15839 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:32.413965 15802 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.514s	user 0.010s	sys 0.007s
I20260709 13:04:32.437857  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1533) took 2164 ms. Trace:
I20260709 13:04:32.418689 15666 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:32.441207  9328 rpcz_store.cc:276] 0709 13:04:30.262213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.285678 (+ 23465us) service_pool.cc:224] Handling call
0709 13:04:30.286055 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.286319 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:30.350968 (+ 64649us) log.cc:844] Serialized 14 byte log entry
0709 13:04:32.305883 (+1954915us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b18003e6f20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:04:32.426062 (+120179us) log.cc:844] Serialized 14 byte log entry
0709 13:04:32.426267 (+   205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:32.426325 (+    58us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:32.426398 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.426492 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.426656 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":68600,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":204,"spinlock_wait_cycles":23985536,"thread_start_us":119631,"threads_started":1}
W20260709 13:04:32.432291 15599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.457s	user 0.017s	sys 0.009s
I20260709 13:04:32.468621 14906 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:32.441542 15605 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:32.466454 15666 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:32.475809  8731 raft_consensus.cc:606] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Received request to transfer leadership to TS d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:32.476234  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.432778 15815 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:32.440970 15795 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.013s	user 0.017s	sys 0.009s
I20260709 13:04:32.429896 15807 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:32.466719 15802 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:32.443687 15400 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:32.492383 15515 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.478436 15666 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:32.465050 15900 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
W20260709 13:04:32.495237 15795 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.071s	user 0.017s	sys 0.016s
I20260709 13:04:32.479835 15605 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.444780 15839 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.510727  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.508229 15886 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:32.525995  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.533039  9326 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:32.563519  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:32.556406  9294 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:32.558970  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:32.567466  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:32.571662  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.572144  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.573020  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.497984 15400 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:32.576856  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 143392us
I20260709 13:04:32.573714  9328 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:32.568912 15667 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:32.588747  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:32.594897  9322 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:32.560673 15777 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.569130 15815 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.606060  8724 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:32.557957 15599 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:32.618455  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.569124 15823 raft_consensus.cc:2764] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:04:32.498571 15807 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.577440 15856 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.598229 15400 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:32.509343 15504 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.635995  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.636715  9331 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:32.622632 15599 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:32.644928  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.634359 15856 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.648772 15856 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.650187  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.650938  9327 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:31.130788  8732 rpcz_store.cc:276] 0709 13:04:29.730742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.730902 (+   160us) service_pool.cc:224] Handling call
0709 13:04:29.731336 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.731633 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:29.737859 (+  6226us) log.cc:844] Serialized 14 byte log entry
0709 13:04:31.075334 (+1337475us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b18004a27e0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b623a 00007f1409c20a2a
0709 13:04:31.128926 (+ 53592us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.128978 (+    52us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:31.129022 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.129085 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.129247 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2325,"prepare.queue_time_us":5954,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":16087,"raft.queue_time_us":1184,"spinlock_wait_cycles":7939968,"thread_start_us":53151,"threads_started":1}
I20260709 13:04:32.622872 15815 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.659482 15815 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.616515 15667 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.677078  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:32.681753  9327 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:32.683564 15599 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.621498 15777 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.697130 15599 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:32.697790 15777 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:32.674872 15802 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.684404 15914 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:32.694247 15815 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:32.718839 15907 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:04:32.731389 15569 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:32.741899 15839 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.746417 15640 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:32.746598 15515 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.748564 15667 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:32.704885 15802 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.751773 15802 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.759843 15504 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.706256 15643 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.762326  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:32.714617 15815 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.710141 15914 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.777340 15823 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.767675 15856 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:32.793860 15719 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:32.778335 15538 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:32.852247 15886 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.875129  8548 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:32.876294 15916 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:32.876945 15911 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 13:04:32.791188 15694 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 13:04:32.820307 15605 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:32.873572  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.885157 15719 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:32.892134  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.779351 15643 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:32.875349  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 95125.5us
I20260709 13:04:32.900774  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:32.892673 15884 raft_consensus.cc:993] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: : Instructing follower d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:04:32.964294  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:32.968911  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66595us
W20260709 13:04:32.970794  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 70104.4us
I20260709 13:04:32.969918  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:32.975946 15729 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.558s	user 0.011s	sys 0.000s
I20260709 13:04:32.976811 15643 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:32.901861 15916 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.978943 15914 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.979291  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:32.980211 15916 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:32.961113 15913 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:32.980993  8582 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:32.982225  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:32.983052  8577 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:32.902403 15795 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.961714 15719 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:32.898550 15687 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:32.884094 15823 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.978974  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:32.885959 15856 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:32.990276  8729 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:32.996169  8581 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:32.993376  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:32.888607 15538 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:32.967414 15663 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:32.981073 15815 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:32.998541  8578 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:33.013147  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:33.023453  8581 raft_consensus.cc:1240] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:33.004828  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 130744us
I20260709 13:04:33.022296  8577 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:33.029718  9330 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:33.026309  8579 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:33.033576  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:33.034523  8584 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 22: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:33.036092  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:33.038026  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.044377  8584 raft_consensus.cc:1240] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:33.047063  8579 raft_consensus.cc:1275] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:33.048540  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:33.052110  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:33.047183  8577 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:04:33.055878  8578 raft_consensus.cc:1275] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260709 13:04:33.003314 15666 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.503s	user 0.007s	sys 0.004s
I20260709 13:04:32.991379 15504 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:33.003979 15851 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.588s	user 0.003s	sys 0.008s
I20260709 13:04:33.060263  8581 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:32.988209 15917 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:33.003979 15569 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:32.997473 15515 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:32.998050 15855 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.022699 15914 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:33.025745 15920 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:33.059777 15694 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:33.064535 15815 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:33.078418  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.004240 15687 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.074433 15888 raft_consensus.cc:1081] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Signalling peer d0a2b9c938d94a4fa258e068ea054d84 to start an election
I20260709 13:04:32.994356 15795 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.031687 15663 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.077541 15914 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:33.079224 15750 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.079428 15920 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:33.068110 15667 consensus_queue.cc:1059] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.062s
I20260709 13:04:33.064891 15666 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.067953 15569 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.067354 15504 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.077347 15515 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.210142  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:33.067545 15851 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.222523  8577 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:33.079779 15855 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.092227 15400 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.204845 15914 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:33.196559 15661 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.196864 15839 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.099361 15694 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.120810 15815 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:33.242642  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 221292us
I20260709 13:04:33.253619  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:33.254753  8577 raft_consensus.cc:2393] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:33.243896  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:33.261303  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.264904  8581 raft_consensus.cc:1275] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:33.212301 15750 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.233901 15802 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:04:33.275888  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.276295  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.276628  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.277657  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:33.282038  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.282905  8725 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:33.283347  8730 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:33.277906  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:33.284282  8726 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:33.278965  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.279764  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.286923  8728 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:33.292919  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:33.295904  9325 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:04:33.247717 15661 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.298122  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:33.301679  8724 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:33.299531  9325 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:33.247499 15839 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.364085  8726 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:33.262391 15920 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.314543 15856 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.368844 15663 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:33.368867 15400 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:33.371416 15922 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 13:04:33.311355 15923 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 13:04:33.367771 15538 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:33.392463 15661 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:33.394701 15913 raft_consensus.cc:3055] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:33.378054 15823 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.404678 15851 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.250393  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:33.389390 15687 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.590211 15729 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.661909 15303 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:33.463626 15776 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.538283  8577 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:33.534125  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:33.526104  8575 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.015s	sys 0.022s
W20260709 13:04:33.515039  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.546316  8658 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:33.697314  8575 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.023s	sys 0.034s
I20260709 13:04:33.531559  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:33.586717  9294 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.65:46779 timed out after 1.618s (SENT) [suppressed 10 similar messages]
W20260709 13:04:33.533694  8580 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.015s	sys 0.017s
W20260709 13:04:33.543540  8650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3486) took 2850 ms (client timeout 1770 ms). Trace:
W20260709 13:04:33.708009  8650 rpcz_store.cc:269] 0709 13:04:30.692843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.693230 (+   387us) service_pool.cc:224] Handling call
0709 13:04:33.542968 (+2849738us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54001d1fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:33.543227 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3402625280}
I20260709 13:04:33.581429 15515 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.713589 15679 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:33.714802 15827 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.714592 15679 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.715438 15687 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:33.715791 15827 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.717861 15890 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.718037  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.718931 15890 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.702740 15636 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:33.720916 15636 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:33.723389 15939 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 13:04:33.723560 15569 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.399904 15913 raft_consensus.cc:740] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:33.724464 15636 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:33.436312 15903 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.838s	user 0.001s	sys 0.009s
I20260709 13:04:33.409260 15719 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.663360 15729 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.729585 15792 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:33.730249 15729 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.730849 15792 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.716683 15687 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.709385  8731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43176 (request call id 54) took 2732 ms. Trace:
I20260709 13:04:33.737412  8731 rpcz_store.cc:276] 0709 13:04:30.977075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.977231 (+   156us) service_pool.cc:224] Handling call
0709 13:04:33.708823 (+2731592us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b5400338954. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c10853 00007f140ebde60a 00007f14081647d4 00007f1408164762 00007f1408164691 00007f140816460c 00007f14081638c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:33.709049 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3444425600}
I20260709 13:04:33.738762  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1549) took 1609 ms. Trace:
I20260709 13:04:33.719489  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1546) took 1153 ms. Trace:
I20260709 13:04:33.740144  9330 rpcz_store.cc:276] 0709 13:04:32.565844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.567124 (+  1280us) service_pool.cc:224] Handling call
0709 13:04:33.718930 (+1151806us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400319054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:33.719165 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":962139392}
I20260709 13:04:33.460718 15914 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.743368 15601 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.744392 15601 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.727286  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1552) took 1100 ms. Trace:
I20260709 13:04:33.739266  9323 rpcz_store.cc:276] 0709 13:04:32.129644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.129786 (+   142us) service_pool.cc:224] Handling call
0709 13:04:32.130144 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:32.134914 (+  4770us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:32.134949 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:32.136597 (+  1648us) log.cc:844] Serialized 26 byte log entry
0709 13:04:32.742319 (+605722us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18004eed40. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:32.992059 (+249740us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:32.992181 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.992361 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:33.707482 (+715121us) raft_consensus.cc:1685] finished
0709 13:04:33.738087 (+ 30605us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:33.738551 (+   464us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4364,"raft.queue_time_us":6082,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":18946,"spinlock_wait_cycles":3878784,"thread_start_us":249356,"threads_started":1,"wal-append.queue_time_us":1025701}
I20260709 13:04:33.580991 15777 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.590251  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.538201  8550 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:33.752339  8580 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.019s	sys 0.029s
I20260709 13:04:33.584663 15504 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.745141 15914 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.759369  8581 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:33.759217 15824 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:33.601912 15814 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:33.761463 15913 consensus_queue.cc:260] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.761974  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.763027 15827 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.434836 15750 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.713009 15889 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:33.418212 15855 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.766100 15917 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:33.414060 15807 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:33.769542 15538 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.769805  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:33.434296 15815 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.459724 15820 consensus_queue.cc:260] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.770843 15777 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.554997  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:33.761255 15890 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:33.589555 15694 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:33.746495  9331 rpcz_store.cc:276] 0709 13:04:32.627070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.640920 (+ 13850us) service_pool.cc:224] Handling call
0709 13:04:33.724504 (+1083584us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:33.725313 (+   809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:33.725345 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:33.725436 (+    91us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:33.725544 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.725683 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:33.726944 (+  1261us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400319050. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:33.727060 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5865,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":328,"spinlock_wait_cycles":2412161408}
I20260709 13:04:33.587199 15666 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.725265 15636 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:33.805068 15827 consensus_queue.cc:1048] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:33.768923 15303 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:33.468537 15795 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.436111 15839 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.795430 15913 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:33.805683 15729 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:33.812369  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.837949 15303 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.837977 15729 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.822265  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:33.841673  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.841603  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.842155 15729 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:33.825801 15636 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.843197  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:33.843899  8729 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:33.844648 15636 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.801681  8726 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:33.772104  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 514966us
W20260709 13:04:33.797827  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.794631 15889 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.811839 15679 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:33.800207 15820 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:33.849080  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 323695us
I20260709 13:04:33.849617 15889 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.809906 15824 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.853821 15569 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:33.854478 15569 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:33.855062 15569 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:33.856240 15824 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.857107 15504 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:33.858449  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:33.799114  8732 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:33.861032  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.862720  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:33.862448 15814 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:33.803853  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 258405us
I20260709 13:04:33.802353 15777 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:33.824895  8697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.839556  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.865286  8725 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:33.867290 15888 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:33.868079 15888 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.839215  8728 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:33.796578  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.870806  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:33.871182 15709 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:33.871590 15610 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:33.876510 15888 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.876919  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:33.877947  9323 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:33.871937  8728 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:33.878809 15538 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:33.880015 15666 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:33.884265  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:04:33.885823  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.886001  8725 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:04:33.888299  9322 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1541) took 1381 ms. Trace:
I20260709 13:04:33.891098 15667 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:33.891624 15667 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:33.892043 15667 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:33.892823 15942 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.129s	user 0.002s	sys 0.000s
I20260709 13:04:33.898592  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:33.898864  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:33.899694  9330 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:33.900208 15920 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:33.900622  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:33.901528  9328 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:33.872818  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:33.852495  8550 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:33.888818  9322 rpcz_store.cc:276] 0709 13:04:32.506629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.509724 (+  3095us) service_pool.cc:224] Handling call
0709 13:04:32.510093 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:32.604527 (+ 94434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.887787 (+1283260us) spinlock_profiling.cc:243] Waited 84.6 ms on lock 0x7b5400867754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:33.888135 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194510080}
I20260709 13:04:33.879877 15687 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:33.933935  9331 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:33.939363  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:33.940200  9327 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:33.958861  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:33.929651 15538 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:33.990919  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 404339us
I20260709 13:04:33.967185 15610 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:04:33.995473  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 143213us
I20260709 13:04:34.005826 15792 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.018376 15705 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:34.019985  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:34.023218 15814 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:34.023772 15950 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.024724 15601 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.027393 15777 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:34.028038 15614 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:34.033331  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 160613us
I20260709 13:04:34.039147 15946 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:34.046119  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 162100us
I20260709 13:04:34.039745 15538 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.035410 15610 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:34.080459  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:34.087350  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 225060us
I20260709 13:04:34.075516 15814 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:34.062161 15777 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:34.093889  8697 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:34.080181 15610 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:34.099503 15948 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:34.099900  9325 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:34.099958 15950 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.100987 15614 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.101732 15719 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.102762 15795 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:34.125769 15795 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.128611 15795 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.105525 15681 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:34.106016 15951 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.091s	user 0.000s	sys 0.002s
I20260709 13:04:34.106379  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:34.150077  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 43842.6us
I20260709 13:04:34.150609  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:34.151283 15666 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:34.151845 15666 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:34.152451 15666 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:34.110025  8724 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.666s	user 0.012s	sys 0.004s
I20260709 13:04:34.113533 15946 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.152779 15882 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:34.153183  8724 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:34.114234  8583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41980 (request call id 3403) took 1106 ms. Trace:
I20260709 13:04:34.120749  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:34.120913  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.117540 15687 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.154996 15946 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.155413 15687 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.154433  8583 rpcz_store.cc:276] 0709 13:04:33.007855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:33.008055 (+   200us) service_pool.cc:224] Handling call
0709 13:04:33.008483 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:33.240303 (+231820us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:33.711808 (+471505us) log.cc:844] Serialized 14 byte log entry
0709 13:04:33.944728 (+232920us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:33.984216 (+ 39488us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:33.993703 (+  9487us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.113739 (+120036us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400179154. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:34.113819 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:34.114030 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53352,"prepare.queue_time_us":458958,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":1823,"raft.queue_time_us":5800,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":17883,"spinlock_wait_cycles":532507392,"thread_start_us":659354,"threads_started":2}
I20260709 13:04:34.062083 15705 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.155037  9331 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:34.089387 15889 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:34.159731 15914 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.160140 15705 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.159649 15889 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.160640 15914 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.162357 15777 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.163965 15914 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:34.164906  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.124480 15950 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.132306  8584 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:04:34.134490 15614 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.135635  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.137223  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:34.137607 15538 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.141077  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.153348 15882 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:34.156052 15643 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:34.103396 15839 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.092631 15601 raft_consensus.cc:476] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Not starting pre-election -- already a leader
I20260709 13:04:34.167091 15815 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.184360  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.182301 15303 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:34.199301  8650 raft_consensus.cc:2468] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:34.198845 15815 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.181070  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.200450 15839 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.181823  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:34.201769  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 109258us
I20260709 13:04:34.181278 15824 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:34.202929 15824 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.203177 15839 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.205068  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.181102 15792 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.206498 15792 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.206842  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.207751  8724 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:34.206112  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:34.208698 15792 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.157585  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:34.095731 15814 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:34.210696  8657 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:04:34.211047 15882 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:34.174119  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:34.210747  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 126088us
I20260709 13:04:34.212687  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:34.129824 15719 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.203277 15889 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.205332 15824 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:34.181939  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61370.7us
W20260709 13:04:34.151898  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 61994.4us
I20260709 13:04:34.114401  9323 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:34.191320  8658 raft_consensus.cc:1275] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:34.216547  9328 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:34.187266  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:34.220381  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.235298  8725 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:34.236189  8652 raft_consensus.cc:3055] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:34.236585  8652 raft_consensus.cc:740] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:34.237506  8652 consensus_queue.cc:260] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.238607  8652 raft_consensus.cc:3060] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:34.239885  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.222185  8582 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:34.223297  8726 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:34.226322  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.228885 15666 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.230141  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.242285  9327 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:04:34.230770 15681 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:34.231101 15538 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:34.251196 15807 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:34.259519 15807 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:34.259871 15807 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:34.263675 15855 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.017s
I20260709 13:04:34.264956 15855 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:34.265384 15855 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:34.266453 15855 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.268198 15855 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:34.232997 15504 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.234249  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.239835 15917 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:34.169708  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:34.221668  8729 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:34.230684  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.246629  8730 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:34.248950  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.250109 15400 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:34.250355 15719 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.251348  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:34.254175  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42702 (request call id 1348) took 1979 ms. Trace:
I20260709 13:04:34.254614 15666 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.255991 15667 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.281349  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.290940 15667 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.291190  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.256867  8659 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 13:04:34.219426 15814 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:34.274040  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.297859  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:34.302135  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.279704 15504 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.168000 15687 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.298051  8656 raft_consensus.cc:1240] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:34.295686  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.288369  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.287403 15719 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:34.297533  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.273173  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:34.293424  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:34.293349  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:34.312038  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 99481.8us
I20260709 13:04:34.296972  8581 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:34.281695  8655 raft_consensus.cc:2468] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:34.318979  8584 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:34.289001  8727 rpcz_store.cc:276] 0709 13:04:32.246517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.247151 (+   634us) service_pool.cc:224] Handling call
0709 13:04:32.247445 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:32.247645 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:32.247672 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:33.764702 (+1517030us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b2c000b2cd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:04:33.765577 (+   875us) log.cc:844] Serialized 26 byte log entry
0709 13:04:33.835474 (+ 69897us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:33.845837 (+ 10363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.914482 (+ 68645us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:34.214670 (+300188us) raft_consensus.cc:1685] finished
0709 13:04:34.214687 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:34.225297 (+ 10610us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3437312,"thread_start_us":69382,"threads_started":1,"wal-append.queue_time_us":164799}
I20260709 13:04:34.319808  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:34.232197  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.322726  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:34.285377 15400 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:34.321532  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 79421.1us
I20260709 13:04:34.321467  8650 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:34.328343  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:34.329063  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.323523 15795 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 13:04:34.334232  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:34.298071 15681 consensus_queue.cc:1048] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:04:34.337185  8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3514) took 2413 ms. Trace:
I20260709 13:04:34.295795 15610 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.340288  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:34.338930  8658 rpcz_store.cc:276] 0709 13:04:31.919566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.919680 (+   114us) service_pool.cc:224] Handling call
0709 13:04:33.545657 (+1625977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:34.225493 (+679836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.333081 (+107588us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54001d1fd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:34.333212 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3738068864}
I20260709 13:04:34.338080  8651 raft_consensus.cc:2410] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 17 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 13:04:34.273777 15948 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.344643  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.345980 15920 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:34.328554 15400 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:34.350843  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:34.340567  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 47287.9us
I20260709 13:04:34.345898  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.328094 15687 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:34.295341 15666 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 13:04:34.300891 15807 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.351542  8652 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 12.
I20260709 13:04:34.283272 15926 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20260709 13:04:34.359341  9326 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:34.360729  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.360239  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.332957  8580 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 11.
I20260709 13:04:34.285987 15569 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.357676  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.367448  8622 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:34.345898 15948 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.351372 15920 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:34.421252  8652 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:34.413053  8659 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:34.421315  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.421922  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:34.365595  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:34.330675 15914 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:34.423523  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 103856us
I20260709 13:04:34.424242  8656 raft_consensus.cc:1275] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:04:34.420132  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:34.334192 15890 consensus_queue.cc:1059] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.030s
W20260709 13:04:34.427811  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.430943  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 72815.7us
W20260709 13:04:34.435019  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.365332 15754 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:34.365540 15720 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:34.416658 15919 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:34.419588 15815 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:34.445994  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 85543.4us
I20260709 13:04:34.452124  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:34.357354 15824 consensus_queue.cc:1048] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.087s
I20260709 13:04:34.456650  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:34.456264  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.458899  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.460096  9331 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:34.456916  8729 raft_consensus.cc:1240] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:34.458900  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.461318  8652 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:34.462853  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:34.465021  9325 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:34.460201  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:34.466763  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:34.467437  8725 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:34.468592  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.466663  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:34.471004  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.471199  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:34.473843  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:34.444139 15709 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:34.475252  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.423415 15719 consensus_queue.cc:1059] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 13:04:34.446462 15610 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.455515 15666 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:34.364966  8476 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 18, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:34.477638  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:34.494266  8726 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:34.479094  8724 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
W20260709 13:04:34.479979  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.483237  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.485512  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.486461  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.452754 15919 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:34.497592  9323 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:34.471630  9327 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 22.
I20260709 13:04:34.459924  8650 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e"
dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
 from {username='slave'} at 127.0.0.1:38104
I20260709 13:04:34.460196 15754 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:34.498687  8650 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:04:34.499053  8650 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:34.493721  8728 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
W20260709 13:04:34.500663  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 66514us
I20260709 13:04:34.468079 15720 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:34.473945 15948 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:34.474298 15504 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.502596  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.509570  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:34.511471  9326 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1545) took 2003 ms (client timeout 2142 ms). Trace:
W20260709 13:04:34.511947  9326 rpcz_store.cc:269] 0709 13:04:32.507955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.563262 (+ 55307us) service_pool.cc:224] Handling call
0709 13:04:34.503472 (+1940210us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54007ab554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:34.511412 (+  7940us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3986702080}
I20260709 13:04:34.478147 15709 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:34.517894  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.512841  8658 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:04:34.502302 15919 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:34.518707  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:34.524083  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.509873  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:34.528306  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:34.530474  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:34.532364  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 180802us
I20260709 13:04:34.509105 15666 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:34.518455 15709 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:34.536564  8654 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:34.537001  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.520804 15807 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.526418 15827 raft_consensus.cc:3055] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:34.540706  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.541157  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.523554 15948 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:34.547092  8654 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:34.547075  8658 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
W20260709 13:04:34.554123  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 107361us
I20260709 13:04:34.553529  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.508517 15777 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:34.564172  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:34.533064 15926 consensus_queue.cc:1059] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
W20260709 13:04:34.568813  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.546819  8476 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 19, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 19 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:34.546765 15827 raft_consensus.cc:740] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:34.568066  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:34.519170 15920 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.575099 15569 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.563161 15601 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:34.567184 15815 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:34.575202 15754 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.575809 15839 raft_consensus.cc:2764] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:04:34.596017 15478 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 13:04:34.604700 15681 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:34.606568 15827 consensus_queue.cc:260] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.603142  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:34.614437  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:34.595502 15824 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:34.578253 15889 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 13:04:34.582896 15601 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:34.610283  8476 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 13, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 13 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:34.582087 15777 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:34.624272 15705 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:04:34.626515  8473 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 306) took 33388 ms (client timeout 15000 ms). Trace:
I20260709 13:04:34.634634 15824 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:34.644032 15601 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:34.647791 15303 consensus_queue.cc:1048] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 13:04:34.651813 15687 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:34.685861 15687 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:34.685964 15946 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:34.687057 15946 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:34.687469 15946 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:34.687702 15643 raft_consensus.cc:3055] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:34.644666 15569 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:34.688230 15643 raft_consensus.cc:740] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:34.688730 15569 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:34.691635 15956 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 13:04:34.692705 15820 raft_consensus.cc:3055] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:34.693158 15820 raft_consensus.cc:740] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:34.694216 15820 consensus_queue.cc:260] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.686329 15687 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:04:34.664981  8473 rpcz_store.cc:269] 0709 13:04:01.233424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.233820 (+   396us) service_pool.cc:224] Handling call
0709 13:04:34.610901 (+33377081us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:27.752681 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:27.753383 (+   702us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:27.753483 (+   100us) write_op.cc:435] Acquired schema lock
0709 13:04:27.753506 (+    23us) tablet.cc:662] Decoding operations
0709 13:04:28.233710 (+480204us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:28.233814 (+   104us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:28.233847 (+    33us) tablet.cc:685] Acquiring locks for 22 operations
0709 13:04:28.235755 (+  1908us) tablet.cc:701] Row locks acquired
0709 13:04:28.235810 (+    55us) write_op.cc:260] PREPARE: finished
0709 13:04:28.236019 (+   209us) write_op.cc:270] Start()
0709 13:04:28.236181 (+   162us) write_op.cc:276] Timestamp: P: 1783602268235948 usec, L: 0
0709 13:04:28.236220 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 13:04:28.373529 (+137309us) log.cc:844] Serialized 8620 byte log entry
0709 13:04:29.116378 (+742849us) op_driver.cc:464] REPLICATION: finished
0709 13:04:29.470724 (+354346us) write_op.cc:301] APPLY: starting
0709 13:04:29.470851 (+   127us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:29.752735 (+281884us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:29.752851 (+   116us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:30.086399 (+333548us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:30.094086 (+  7687us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=44,key_file_lookups=44,delta_file_lookups=44,mrs_lookups=0
0709 13:04:30.094163 (+    77us) write_op.cc:312] APPLY: finished
0709 13:04:30.145240 (+ 51077us) log.cc:844] Serialized 196 byte log entry
0709 13:04:30.390815 (+245575us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:30.423343 (+ 32528us) write_op.cc:454] Released schema lock
0709 13:04:30.461643 (+ 38300us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:30.515496 (+ 53853us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:30.520015 (+  4519us) write_op.cc:454] Released schema lock
0709 13:04:32.324429 (+1804414us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
Metrics: {"mutex_wait_us":4046389,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":317769,"apply.run_cpu_time_us":76649,"apply.run_wall_time_us":1019998,"cfile_cache_hit":90,"cfile_cache_hit_bytes":96043,"num_ops":22,"prepare.queue_time_us":825217,"prepare.run_cpu_time_us":22574,"prepare.run_wall_time_us":945033,"raft.queue_time_us":298463,"raft.run_cpu_time_us":43810,"raft.run_wall_time_us":324289,"replication_time_us":846605,"spinlock_wait_cycles":27278080,"thread_start_us":1441729,"threads_started":5,"wal-append.queue_time_us":588890,"wal-append.run_cpu_time_us":131990,"wal-append.run_wall_time_us":4295086}]]}
I20260709 13:04:34.691642 15569 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.719799 15303 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:34.721352  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:34.646178 15705 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:34.630136 15948 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.635326 15815 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.708290 15777 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.689220 15643 consensus_queue.cc:260] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.710327 15827 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:34.724562  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 76665.1us
I20260709 13:04:34.725833  8476 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 14, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 14 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:34.728861  8659 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:04:34.596376 15720 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:34.749758  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.683444 15824 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:34.754171  8725 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:34.754473  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:34.632661 15666 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:34.768620  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.667338 15950 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.088s
W20260709 13:04:34.810647  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 293411us
I20260709 13:04:34.755450 15795 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:34.688768 15946 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:34.812711  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 348458us
I20260709 13:04:34.830201 15303 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
W20260709 13:04:34.813268  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 367349us
I20260709 13:04:34.848548 15946 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:34.848491 15795 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:34.851415 15795 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.851568 15303 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 13:04:34.813704  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 250050us
W20260709 13:04:34.893595  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.864943  8476 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 14, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 14 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:34.898056  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.898239  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:34.905589  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.907968  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.909220  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.909482  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.909993  8576 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:34.910030 15478 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:34.910952  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.910912 15478 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.911834  8582 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:34.911923 15705 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.910187  8580 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:34.911440  8576 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 21 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:34.918010  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.919059  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.919243  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:34.920908  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 103535us
W20260709 13:04:34.924899  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 56524.2us
I20260709 13:04:34.927361  8579 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 20: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:34.913223 15478 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:34.932632 15754 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:34.933262 15754 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:34.935566 15754 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:34.924145  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.890908 15504 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.921856  8625 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:34.918426  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.929080  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:34.942489  8476 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:34.948799  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44447.7us
I20260709 13:04:34.950333  8578 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:34.900090 15966 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:34.914325 15614 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:34.973665  8577 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:34.965337  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.926429 15705 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:34.975306  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.960412  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:34.981017  8581 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 13.
I20260709 13:04:34.975579  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:34.943562 15643 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:34.988463  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44025us
I20260709 13:04:34.988607  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:34.994122  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:34.994974  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:34.991791 15478 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:34.985471 15719 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:35.001374  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.001529 15719 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:35.002555  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 80826.6us
I20260709 13:04:35.004030 15719 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.004565  8728 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:04:35.006925 15400 consensus_meta.cc:287] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.640s	user 0.008s	sys 0.003s
I20260709 13:04:35.006515 14906 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 13:04:34.992026 15729 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:34.990274 15504 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.007885 15709 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:35.010845  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.012946  8476 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 14, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 14 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:35.017866  8625 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:35.018651  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:35.023130  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:34.971284  8650 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:34.973603 15614 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:34.993435 15569 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:04:35.072783  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54211.7us
I20260709 13:04:35.073189  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:35.074357  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.044077  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:35.087598  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 64579.8us
I20260709 13:04:35.090226  9330 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 13:04:35.042330 15729 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:35.029021 15914 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:35.094594  8575 raft_consensus.cc:1240] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 10. Current term is 11. Ops: []
I20260709 13:04:35.044406 15719 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:35.095692  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:35.069782 15478 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:35.106514 15478 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:34.983582 15966 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.103374  8698 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:35.107045  8476 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:35.070652 15569 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:35.104552  8732 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:04:35.119306  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:35.120183  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.122957 14906 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:35.008682 15601 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.100195 15729 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:35.042605 15400 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.107759 15795 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:35.100606 15914 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:35.107019 15478 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.105644 15792 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:04:35.124099 15855 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.815s	user 0.008s	sys 0.005s
I20260709 13:04:35.105234 15946 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:35.148685  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 104680us
I20260709 13:04:35.106060 15719 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:35.116366 15839 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:35.134495 15569 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.154276  8652 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:04:35.156817  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.135807 15815 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:35.155412  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:35.151962 15914 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.174497 15614 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.176340 15504 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.154287 15792 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:35.189929  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 100359us
W20260709 13:04:35.208665  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.185716  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.158838 15946 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:35.165992 15719 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:35.170907 15855 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:35.253116  8575 raft_consensus.cc:2410] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
W20260709 13:04:35.235971 15814 consensus_meta.cc:287] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.915s	user 0.014s	sys 0.000s
I20260709 13:04:35.156641 15807 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
W20260709 13:04:35.265686  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 142291us
I20260709 13:04:35.266443  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:35.239830 15839 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:35.251407 15792 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:35.259714  8476 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 10, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 10 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:35.253039 15946 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:35.270246  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:35.276677  8577 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 13:04:35.275120  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 120319us
W20260709 13:04:35.284861  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.283586  8476 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 6 to 13, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 13 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:35.271775 15839 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:35.293064  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 275256us
I20260709 13:04:35.293501  8581 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.294019  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:35.272101 15855 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:35.323256  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:35.290411 15827 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.000s
I20260709 13:04:35.265547 15814 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:35.338747  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47591.5us
I20260709 13:04:35.283318 15807 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.338454  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.342484  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.293315 15839 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:35.351716  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57779.2us
W20260709 13:04:35.351778  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56743.7us
I20260709 13:04:35.281162 15974 consensus_queue.cc:1059] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.067s
I20260709 13:04:35.352878  8476 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:35.360093  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.353281 15997 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:35.361035 15989 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.201s	user 0.000s	sys 0.000s
W20260709 13:04:35.362126 15993 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 13:04:35.362455 15966 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:35.363921 15983 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:35.364333 15995 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:35.364648 16000 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:35.369555 15980 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:35.370400 15601 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.374922 15729 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.387100 15963 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:35.387428  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.347421 15709 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:35.281697 15754 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:04:35.403610  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 235117us
W20260709 13:04:35.475504  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 326542us
I20260709 13:04:35.529872  8577 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:35.442453  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:35.580219  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.575518  8575 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:35.555016  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
W20260709 13:04:35.415932 15987 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260709 13:04:35.422020 15687 consensus_meta.cc:287] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.710s	user 0.012s	sys 0.000s
W20260709 13:04:35.415690 15990 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 13:04:35.440482 15986 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.273s	user 0.001s	sys 0.000s
W20260709 13:04:35.528090 15919 consensus_meta.cc:287] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.993s	user 0.010s	sys 0.002s
W20260709 13:04:35.427863 15882 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.216s	user 0.010s	sys 0.002s
I20260709 13:04:35.396792 15400 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:35.615280  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.600374  8728 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:35.601343 15729 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:35.594173 15974 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 13:04:35.610888  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:35.612336  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:35.619297  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.598412  8476 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 17, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:35.529425 15754 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:04:35.625356  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.625586  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:35.626569  8476 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 8 to 14, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 14 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:35.634439  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:35.637642  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.640043  8725 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:35.639621  8577 raft_consensus.cc:2393] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:35.652077  8725 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 13:04:35.654021  8581 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.653100  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:35.652541  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44577.5us
I20260709 13:04:35.655514  9328 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:35.592530 15963 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.611768 15946 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.658313 15963 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:35.658944 16010 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.659525 16010 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:35.665940 16002 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 13:04:35.666680 15719 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.667303 15890 consensus_queue.cc:1048] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:35.644260 16000 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 13:04:35.670560 16000 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:04:35.670747 15795 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.000s
I20260709 13:04:35.671007 16000 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:35.652984  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3534) took 1419 ms. Trace:
I20260709 13:04:35.568295 15995 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.652760 15827 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
W20260709 13:04:35.674393  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.676285  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.416214 15855 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.677687 15719 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:35.678004  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.678747  8577 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:35.672003 16000 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.697602  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:35.697930 16000 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:35.672185 15820 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:35.698240 15980 consensus_queue.cc:1175] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1dc41db601fe4b91aa6d461b504da0e7" responder_term: 17 status { last_received { term: 17 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 17 index: 11 } }
I20260709 13:04:35.555981 15569 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.682214  8650 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:35.682941  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.684525 15814 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:35.701546 15993 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:04:35.702630  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.703261  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.703970  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.704571  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.705502 15303 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:35.706032 15303 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:35.706228  8731 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:35.646188 15709 consensus_queue.cc:1048] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.062s
I20260709 13:04:35.708344  8583 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
W20260709 13:04:35.688946 15478 consensus_meta.cc:287] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.531s	user 0.006s	sys 0.002s
I20260709 13:04:35.703966 15754 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.710275 15754 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:35.709824 15478 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.712603 15478 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:35.690847  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
W20260709 13:04:35.713644 15987 log.cc:927] Time spent T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 13:04:35.713990  8581 raft_consensus.cc:2393] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:35.713896  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.715374  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3552) took 1445 ms. Trace:
W20260709 13:04:35.715583  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 70231.1us
I20260709 13:04:35.697171 15995 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:35.715847  8655 rpcz_store.cc:276] 0709 13:04:34.270186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.512783 (+242597us) service_pool.cc:224] Handling call
0709 13:04:34.513184 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:34.521249 (+  8065us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:34.842568 (+321319us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.067965 (+225397us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:35.093623 (+ 25658us) log.cc:844] Serialized 26 byte log entry
0709 13:04:35.093761 (+   138us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:35.103101 (+  9340us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.275096 (+171995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.275175 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.715129 (+439954us) raft_consensus.cc:1685] finished
0709 13:04:35.715151 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.715305 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25797,"prepare.queue_time_us":351019,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":106,"raft.queue_time_us":86260,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":105421,"thread_start_us":582770,"threads_started":3,"wal-append.queue_time_us":392481}
W20260709 13:04:35.698042 15824 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.883s	user 0.009s	sys 0.003s
I20260709 13:04:35.722057 15824 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.658468 15946 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.697858  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.622731 15882 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.725067 15995 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:35.725907  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 70151.9us
I20260709 13:04:35.725765 15995 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.726816 15882 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:35.622969 15919 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.728236  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: true dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.728864  9330 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 13:04:35.729157 15919 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.729310  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.727869  8577 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 11.
W20260709 13:04:35.696208 15705 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.704s	user 0.010s	sys 0.000s
I20260709 13:04:35.730291  8730 raft_consensus.cc:2393] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:35.613621 15687 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.728873  8575 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:35.732453 15687 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.733019  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.733897  9330 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:35.734078  9322 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 18: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:35.707069 15303 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.635069 16008 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.087s
I20260709 13:04:35.729735  9327 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 13:04:35.737742  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 787930us
I20260709 13:04:35.737957 16010 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.738749  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:35.701532  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:38104 (request call id 3541) took 1473 ms. Trace:
I20260709 13:04:35.739532 15963 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:35.753890  8650 rpcz_store.cc:276] 0709 13:04:34.228384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.440880 (+212496us) service_pool.cc:224] Handling call
0709 13:04:35.682082 (+1241202us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b4c0045a340. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1407ee2a20 00007f1407ef3a29 00007f1407ef413f 00007f1407eb82ab 00007f1407ee5f3c 00007f1407ee63fe 00007f140817388b 00007f1409b72495 00007f1409bc330f 00007f1409c0da55 00007f140ebddcd0
0709 13:04:35.701367 (+ 19285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10656512}
I20260709 13:04:35.733036 15824 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.755424 15709 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:35.756155  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.691709 15792 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.757304  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:04:35.757791  8476 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 12, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 12 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:35.758983 15792 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.745867  8581 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:35.760708  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:35.760844  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:35.761869  8732 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:35.671778  8657 rpcz_store.cc:276] 0709 13:04:34.226242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.229491 (+  3249us) service_pool.cc:224] Handling call
0709 13:04:34.229862 (+   371us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:34.247110 (+ 17248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:34.247147 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:34.284687 (+ 37540us) log.cc:844] Serialized 52 byte log entry
0709 13:04:34.482361 (+197674us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b1800291400. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:34.797734 (+315373us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:34.798130 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 13:04:34.798485 (+   355us) log.cc:844] Serialized 14 byte log entry
0709 13:04:34.798647 (+   162us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.798746 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.604718 (+805972us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x7b2400220ec8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f14069af9c4 00007f14069adacb 00007f1406b68dc9 00007f1406b68a55 00007f1406b67fb0 00007f14097de488 00007f14097debe8 00007f14097b6398 00007f14097b176a 00007f14097b0bb6
0709 13:04:35.644815 (+ 40097us) raft_consensus.cc:1685] finished
0709 13:04:35.644848 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.645072 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46471,"prepare.queue_time_us":306464,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":112260,"raft.run_cpu_time_us":444,"raft.run_wall_time_us":12133,"spinlock_wait_cycles":147456640,"thread_start_us":315063,"threads_started":1,"wal-append.queue_time_us":512015}
I20260709 13:04:35.599824 15807 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.762828  8728 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 13:04:35.735983 15303 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:35.751904  8622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.765705  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.769907  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.770347  9327 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:35.770658  9330 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 13.
I20260709 13:04:35.771358  8655 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:35.771458  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.772284  8657 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:35.772524  8732 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:35.774893  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.775736  8581 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:35.776155 15997 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:35.776562  8584 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:35.777119  8652 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.777293 15914 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.779583  8650 raft_consensus.cc:1240] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.748560  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.780947  8582 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.765480  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:04:35.784061  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.786010  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.786607 15807 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:35.789626  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.790488 15614 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:35.794636 15687 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:35.795342  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:35.796027  8726 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 13.
W20260709 13:04:35.799531 15973 log.cc:927] Time spent T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.770s	user 0.005s	sys 0.001s
W20260709 13:04:35.802445 16006 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.146s	user 0.002s	sys 0.000s
I20260709 13:04:35.806618  8476 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 6 to 16, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 16 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:35.813524  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.813898 16003 raft_consensus.cc:3055] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:35.815769  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.814344 16003 raft_consensus.cc:740] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:35.817104 15569 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.818002 16003 consensus_queue.cc:260] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.819360 16003 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:35.819337  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.821389  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.822979  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.823639  9330 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 11.
I20260709 13:04:35.825165  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.814877  8575 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:35.825632  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.826400  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:35.827062  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.827307  9327 raft_consensus.cc:1240] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:35.743966  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.782296 15995 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:35.834292  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.839296  8581 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:04:35.840829  9331 raft_consensus.cc:2410] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:04:35.844810  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.825543  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1360) took 1377 ms. Trace:
I20260709 13:04:35.847386  9328 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:35.856314  9323 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:35.787899 15914 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:35.848661  8730 rpcz_store.cc:276] 0709 13:04:34.448053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.448224 (+   171us) service_pool.cc:224] Handling call
0709 13:04:35.825313 (+1377089us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400203fd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:35.825472 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2912304896}
I20260709 13:04:35.890148  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.752010 15839 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:35.890885  8575 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 15.
I20260709 13:04:35.808614  8476 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:35.868745  8582 raft_consensus.cc:2468] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:35.901126  8582 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:35.910506  8580 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.916512  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.921242  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:35.923328  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:35.925987  9328 raft_consensus.cc:1240] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 13:04:35.929931  8476 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:35.932466  8650 raft_consensus.cc:2410] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:35.933084  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:35.935087  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:35.939172  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:35.937426 14906 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:35.938458  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:35.913882 15920 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.941059 15920 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:35.941581  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:35.942456  9330 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:35.943184 15920 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:35.943576  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:35.943871  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:35.944303  9330 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.946044  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:35.947652  8548 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:35.944443  9322 raft_consensus.cc:3060] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:35.950088  8582 raft_consensus.cc:2410] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:04:35.954155  8476 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 17. New cstate: current_term: 17 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:35.960143  8623 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:35.973572  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 363450us
I20260709 13:04:35.969795  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:35.971287  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:35.979946  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:35.987397  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:35.988742  8576 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 14 LEADER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:35.990808  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:35.983234  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:35.997443  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:35.998085  8726 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.988206  8730 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:36.001916  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:36.002830  8582 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:36.003162  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.009860  8476 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 15, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 15 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:36.013001  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:36.014087  9328 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:36.051956  8731 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 13:04:36.038335  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.055583  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.055565  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 64833.1us
I20260709 13:04:36.056756  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:36.064033  8728 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:36.057641  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:36.057819  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:36.021368  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.067615  8726 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:35.968581 16003 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:36.069557  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.070467  8654 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:36.070000  8476 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 15, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 15 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:36.076432  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.110031  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:36.110867  8658 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 23: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:36.111528  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.001236 15666 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:36.097877  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:36.100835  8659 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:36.002689 15792 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.102245  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:36.104390  9330 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:35.969127 15914 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:36.017303 15855 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:35.994557 15815 raft_consensus.cc:2764] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:04:36.277249  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:35.977380 15754 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 13:04:36.115098  9324 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.994987 15614 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:36.016852 15926 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:35.977849 16008 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.120s
I20260709 13:04:36.120152  9323 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:36.139627  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:36.158648  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.155721  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 147487us
W20260709 13:04:36.163556  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 110037us
I20260709 13:04:36.287747  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.001500 15400 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.113s
I20260709 13:04:36.056608 15920 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 13:04:36.196122  8578 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:36.192699  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:36.296314  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:36.199633  8476 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 16, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 16 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:36.115592 15666 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:36.274014  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:36.303484  9329 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:36.194324 15914 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:36.306054  8656 raft_consensus.cc:2393] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 12: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:36.279671  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 87282.4us
I20260709 13:04:36.280548  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.280189  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.100117 16003 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.308214  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:36.140347 15792 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.308023  8653 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
W20260709 13:04:36.281183  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:36.293274  9326 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:36.294210  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:36.295207 15705 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:36.295887  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.313603  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:36.313087  8476 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 7 to 17, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 17 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:36.314373  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:36.280102 15992 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.281349 14906 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.291940  8659 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:36.291535  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:36.308606  9294 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:36.314296  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.315866 14906 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.324661  8654 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:36.280242 16016 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:36.317904  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:36.336709  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.338656  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.341554  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.343616  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.347086  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.306733 15666 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:36.301683 15991 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:36.313648 15855 raft_consensus.cc:3055] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:36.352818  8730 raft_consensus.cc:2368] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20.
I20260709 13:04:36.353729  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.359637  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:36.359928  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.306994 15926 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 20 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:36.351495  8656 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 22.
I20260709 13:04:36.288251 15614 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.366247  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.309798 15890 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.309556 15914 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:36.347855 15400 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:36.377123  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:36.370266  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.370476  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.370976  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.374300  8659 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
W20260709 13:04:36.349123 15303 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.585s	user 0.009s	sys 0.001s
I20260709 13:04:36.413553  8576 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:36.414659 15926 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.416039  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.448333  8656 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 12.
I20260709 13:04:36.350090  8725 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 13: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:36.424849  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.428443  8733 raft_consensus.cc:2410] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:04:36.360493 15995 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.451472  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.452135  8658 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:04:36.447644  8476 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 9 to 12, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 12 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:36.367143 16010 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:36.467885  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.468761  8659 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
I20260709 13:04:36.469741  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:36.325289 15705 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.373463 15855 raft_consensus.cc:740] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:36.425716 15400 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:36.345651 15992 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.348410 15815 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.101s
I20260709 13:04:36.485939  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.486979  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.504312  8656 raft_consensus.cc:2410] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:36.478703  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.501992 14906 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:36.505296  8724 raft_consensus.cc:2393] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 21: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:36.512560  9324 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:36.514680  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.527379  8659 raft_consensus.cc:2410] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:04:36.471215 15687 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:36.522873  8578 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:36.529091  9323 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260709 13:04:36.419577 15963 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 13:04:36.546972  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.548362  8656 raft_consensus.cc:2410] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
W20260709 13:04:36.546957  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:36.496517 15729 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.503618 15400 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:36.491365 16010 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:36.449263 15890 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.571970  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.573136  8724 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:36.513991 15538 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:36.561153 16018 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.283s	user 0.000s	sys 0.001s
W20260709 13:04:36.581606  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 238536us
I20260709 13:04:36.587986  8658 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:36.577297  9324 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:36.479321 15995 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.592396  8696 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.592866  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.582803  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:36.608023 15889 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:36.616010  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 251262us
I20260709 13:04:36.575642 15991 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:36.597958 16014 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.407s	user 0.001s	sys 0.000s
I20260709 13:04:36.625836  8725 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:36.628599  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:36.642422  9322 consensus_meta.cc:287] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.668s	user 0.010s	sys 0.001s
I20260709 13:04:36.642290  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.642926  9322 raft_consensus.cc:2468] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 12.
I20260709 13:04:36.561720 15966 consensus_queue.cc:1059] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
W20260709 13:04:36.645153  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 294366us
I20260709 13:04:36.574813 15729 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.651643  9324 raft_consensus.cc:1275] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:36.648767  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.601449 15687 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:36.597532 15985 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:36.651373  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.655375  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.650194  8623 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:36.659219  8658 raft_consensus.cc:2468] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:36.683583  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.585693 15538 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:36.660866 16013 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.363s	user 0.001s	sys 0.000s
I20260709 13:04:36.695640  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:36.696139  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 155209us
I20260709 13:04:36.646394 16016 raft_consensus.cc:697] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:36.700511  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 50909.1us
W20260709 13:04:36.695513 16022 log.cc:927] Time spent T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.395s	user 0.001s	sys 0.000s
W20260709 13:04:36.692065 16024 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.323s	user 0.001s	sys 0.000s
I20260709 13:04:36.681118 15303 raft_consensus.cc:2804] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:36.662593 16003 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:36.717564  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.641578 15991 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:36.740721 16028 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 13:04:36.741191  8726 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:36.741012  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.744310  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: true dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:36.731895 16014 log.cc:927] Time spent T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 13:04:36.710592 16021 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:04:36.764122 15991 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:36.735234 15614 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:36.755761 16008 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:36.767027  9296 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:36.768391 15992 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:36.682353 15985 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:36.769511 16023 log.cc:927] Time spent T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 13:04:36.714288 16033 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:04:36.747845  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.710772 16010 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.749210  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.751847  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.759475  8658 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:36.614843 15855 consensus_queue.cc:260] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.779522  9329 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:36.781208  9328 raft_consensus.cc:1240] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:36.760686  8728 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.649s	user 0.010s	sys 0.001s
I20260709 13:04:36.709348 15729 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:36.761574  8733 raft_consensus.cc:3055] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:36.763358 15538 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:36.773916  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:36.775485  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.714206 15890 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:36.810292  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.824816  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 111922us
I20260709 13:04:36.778416  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:36.807610 16029 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 13:04:36.808825  8733 raft_consensus.cc:740] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:36.791589  8728 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:36.760306 15303 raft_consensus.cc:697] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:36.825381  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 131088us
W20260709 13:04:36.819180  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.000s	sys 0.012s
W20260709 13:04:36.840179  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.001s	sys 0.012s
W20260709 13:04:36.840520  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 724996us
W20260709 13:04:36.818739  8650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.006s	sys 0.006s
W20260709 13:04:36.799155  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 192007us
W20260709 13:04:36.841683  8650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.006s	sys 0.007s
I20260709 13:04:36.842056  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:36.815683 16047 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:36.847157  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.848402  8658 raft_consensus.cc:2410] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 13:04:36.848924  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:36.793526 16008 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 13:04:36.825657 15729 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
W20260709 13:04:36.839363  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 90263.7us
I20260709 13:04:36.852046  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:36.852459  8549 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:36.853768  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:36.710968 16016 consensus_queue.cc:237] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.843053  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:36.850540  8729 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:04:36.853965  8726 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:36.864244  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:36.864459 15991 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:36.864924 15991 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:36.865260 15991 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:36.865490  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.866075  8658 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:36.868057 15914 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.841632 15705 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:36.861876 16047 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:36.859009  8733 consensus_queue.cc:260] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.869804 15985 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:36.869807 16047 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:36.870443 16047 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:36.871341 15400 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:36.870450  8733 raft_consensus.cc:3060] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:36.845817  8659 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:36.827401  8652 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:36.856395  8696 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:36.856990 16030 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:36.857976  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:36.821218 16049 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:36.892354  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.892570  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:36.893826  8652 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:04:36.894541  8659 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 19
W20260709 13:04:36.899638  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 132676us
I20260709 13:04:36.904507  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.906277  8658 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
W20260709 13:04:36.907068  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 54716.4us
I20260709 13:04:36.870476 15985 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:36.910249  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:36.909375  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.911568 16036 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 13:04:36.891122 16049 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:36.899025 15400 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:36.939021  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82744.8us
I20260709 13:04:36.941712  9296 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:36.942246  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.942648  8659 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:36.943961  8659 raft_consensus.cc:1240] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 16. Current term is 17. Ops: []
W20260709 13:04:36.944523  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 95662.6us
I20260709 13:04:36.926085 16030 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:36.926335 15538 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:36.935740 16031 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 13:04:36.926515 16054 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:36.958994  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.936762 15985 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:36.931813 15995 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:36.962714  9294 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:36.860406 16046 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.102s
I20260709 13:04:36.915073 15729 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:04:36.946843 16036 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:36.938987 16049 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:36.974938  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.974938  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:36.979276  8733 raft_consensus.cc:2468] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:36.963366 16030 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:36.988966  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46769.3us
I20260709 13:04:36.990253  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:36.987952 15985 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:36.981237 16036 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:36.980911 15400 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:36.986420 16041 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:36.999100 16056 consensus_queue.cc:1059] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:36.966915 15890 consensus_queue.cc:1059] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 13:04:37.000579 15995 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:37.000941  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.001579 16043 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:04:37.002177 15666 consensus_meta.cc:287] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.000s
I20260709 13:04:37.003536 15538 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:36.950820 15303 consensus_queue.cc:237] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.030963  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1570) took 1089 ms. Trace:
I20260709 13:04:36.990486 16030 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:36.996505 15729 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:37.013972 16036 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:36.995397 16052 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 13:04:37.030954  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 56098.3us
I20260709 13:04:37.031808  9330 rpcz_store.cc:276] 0709 13:04:35.941515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.124624 (+183109us) service_pool.cc:224] Handling call
0709 13:04:36.194741 (+ 70117us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.194976 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.195000 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:36.281717 (+ 86717us) log.cc:844] Serialized 26 byte log entry
0709 13:04:36.340998 (+ 59281us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:36.341131 (+   133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:36.341345 (+   214us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.030654 (+689309us) raft_consensus.cc:1685] finished
0709 13:04:37.030689 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.030881 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":39253,"threads_started":1,"wal-append.queue_time_us":314930}
I20260709 13:04:37.038086  8650 raft_consensus.cc:1240] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:37.029764 16057 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:37.025230 15995 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.036221  8659 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 13:04:37.040215  8659 pending_rounds.cc:85] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 9
I20260709 13:04:37.034319 16043 raft_consensus.cc:3055] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:37.023681 15400 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.041630  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.049893  8650 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:37.038527 15729 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.040537  8659 pending_rounds.cc:107] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 13:04:37.039888 16054 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:37.041136 15538 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:37.043653 15995 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:37.057286  8659 raft_consensus.cc:2889] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:37.058636  8696 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:37.042199 16043 raft_consensus.cc:740] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:37.031734 15666 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.061169 16057 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:37.073941  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 73070.7us
I20260709 13:04:37.096057  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:37.073928  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 83788.9us
I20260709 13:04:37.097478  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.102308  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:37.093111  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 151498us
I20260709 13:04:37.056869 15729 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:37.104936  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:37.109565 16063 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:37.110188 16058 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.126s	user 0.003s	sys 0.000s
I20260709 13:04:37.115643  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3565) took 1008 ms. Trace:
I20260709 13:04:37.116120 15914 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.116132  8652 rpcz_store.cc:276] 0709 13:04:36.106958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.905570 (+798612us) service_pool.cc:224] Handling call
0709 13:04:36.906014 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.916553 (+ 10539us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.916603 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:36.917249 (+   646us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.014865 (+ 97616us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:37.014933 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.015043 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.115402 (+100359us) raft_consensus.cc:1685] finished
0709 13:04:37.115426 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.115578 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":202,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":1840,"thread_start_us":58876,"threads_started":1,"wal-append.queue_time_us":99115}
I20260709 13:04:37.094544 16056 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:37.127300  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:37.120170  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.109450 16053 consensus_queue.cc:1059] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.096s
W20260709 13:04:37.122373  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 242894us
I20260709 13:04:37.103497 16052 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:37.130508  8696 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.138015  8696 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.144419  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.120085 16054 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:37.123023 15538 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:37.146615  8578 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:37.147281  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:37.148386  8577 raft_consensus.cc:1240] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 15. Current term is 16. Ops: []
W20260709 13:04:37.149611 16007 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.986s	user 0.003s	sys 0.011s
W20260709 13:04:37.150031 16045 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 13:04:37.151226 15814 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:37.151643 16065 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:37.126587 16056 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:37.157361  8576 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 20
W20260709 13:04:37.156826  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:37.160560  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64608.2us
I20260709 13:04:37.175971  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:37.177698 16063 raft_consensus.cc:2764] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:04:37.136636 15815 raft_consensus.cc:2804] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:37.178807 15815 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:37.179333 15815 raft_consensus.cc:3060] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:37.163947  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.181790  8582 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:37.145004 15890 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:37.177153  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.184785 16063 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:37.191196  8609 heartbeater.cc:646] Failed to heartbeat to 127.7.78.126:44983 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.7.78.126:44983 timed out after 15.000s (SENT)
I20260709 13:04:37.195880 16056 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:37.131733 15882 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.031s
I20260709 13:04:37.156771 15666 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.158439 15400 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.159574 15538 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:37.206509 14906 raft_consensus.cc:3055] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:37.213673 14906 raft_consensus.cc:740] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:37.207562 15997 consensus_queue.cc:1048] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:37.208647 16065 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.221495 16065 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.224133 16065 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.228158 15729 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.216882  8696 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:37.232167  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:37.233265 15882 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:37.233745 15882 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.234114 15882 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:37.234217 14906 consensus_queue.cc:260] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.235548 16060 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:37.235941 14906 raft_consensus.cc:3060] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:37.235993 16060 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.236703 16060 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:37.153705 16043 consensus_queue.cc:260] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.238369 16043 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:37.239352 15729 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.241999 15815 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.245848  8576 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:37.257198 15914 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:37.257867 15914 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.258704 16010 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:37.259238 16010 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.260109 15914 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.261415 16010 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:37.262202  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.263775  8582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.264289 15920 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.264616  8582 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:37.264802 15920 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.263825  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.266309 16039 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:37.265221 15839 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.267002 15920 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:37.267481  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:37.267735  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.268368  8656 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:37.269790  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:37.270268  8654 raft_consensus.cc:1240] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:37.270922  8548 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:37.271642  8653 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 13:04:37.271992  8653 pending_rounds.cc:85] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 14
I20260709 13:04:37.272193  8653 pending_rounds.cc:107] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 13:04:37.272392  8653 raft_consensus.cc:2889] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:37.273759 15966 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.274253 15966 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.276348 15966 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.248070 15815 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.278537 15815 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:37.279239 16052 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:37.279729 16052 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:37.280570 16052 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.281363 15839 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.282951 15839 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.284613  8651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3558) took 1262 ms. Trace:
I20260709 13:04:37.282147  8698 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:37.252555 15882 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.257871 15985 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:37.258085 16059 raft_consensus.cc:3055] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:37.291086  8622 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:37.251446 16047 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.264642  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.298880  8581 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.301304  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.303546  8698 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.305207  8698 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.285179  8651 rpcz_store.cc:276] 0709 13:04:36.021952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.023866 (+  1914us) service_pool.cc:224] Handling call
0709 13:04:36.070137 (+ 46271us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.471229 (+401092us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.471273 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:36.488451 (+ 17178us) log.cc:844] Serialized 26 byte log entry
0709 13:04:36.606059 (+117608us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:36.606100 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.284094 (+677994us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b540007dfd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:37.284165 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.284233 (+    68us) raft_consensus.cc:1685] finished
0709 13:04:37.284250 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.284536 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":921668352,"thread_start_us":104279,"threads_started":1,"wal-append.queue_time_us":146813}
I20260709 13:04:37.308219  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:37.310842  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.311916  8581 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 22.
I20260709 13:04:37.313071  8579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.314652  8583 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.282336 16057 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:37.216840 15538 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:37.288919 15814 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:37.327615  8584 raft_consensus.cc:1240] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:04:37.333766  8580 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.286556 15815 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:37.329958  8622 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:37.339761  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.339844  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.343101  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.294791 16059 raft_consensus.cc:740] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:37.308559 15839 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.307612 16039 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:37.308401 15890 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:37.307088 16054 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:37.310461 15795 raft_consensus.cc:2804] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:37.356604  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.361953  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.365046  8578 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.356719 16035 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.369017  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:37.369751 16039 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:37.370146  8576 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 14: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:37.371320  8576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.371827 16037 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:37.373219 15754 raft_consensus.cc:3055] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:37.373684 15754 raft_consensus.cc:740] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:37.362802 15995 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.374655 15754 consensus_queue.cc:260] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.375947 15754 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:37.376027 15995 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.332450 15814 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:37.326787 16057 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:37.350553 15705 raft_consensus.cc:2804] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:37.325922 15985 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.309880 16060 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.408644  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:37.369199 16035 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.398108 16057 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:37.413331  8578 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:37.393965 15795 raft_consensus.cc:697] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:37.419317  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 187218us
I20260709 13:04:37.401484 16050 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:37.424568  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:37.425813  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:37.371073 16054 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.432605  9322 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:37.410336 15705 raft_consensus.cc:697] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:37.409914 15729 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:278: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
I20260709 13:04:37.409662 15995 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:37.440791  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:37.441895  9326 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:37.442704  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:37.442772  8548 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:37.393174 15991 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.445742  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:37.448894  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:37.451787  9329 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:04:37.452301  9329 pending_rounds.cc:85] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Aborting all ops after (but not including) 11
I20260709 13:04:37.452569  9329 pending_rounds.cc:107] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:04:37.452903  9329 raft_consensus.cc:2889] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:37.454732  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:37.456677  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48080us
I20260709 13:04:37.436208 15926 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:37.370237 16059 consensus_queue.cc:260] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.463914  9325 raft_consensus.cc:1240] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:04:37.474960  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
/apache/dev/git/kudu/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
W20260709 13:04:37.498417  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 184806us
I20260709 13:04:37.478895  8576 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 20. Ops: []
I20260709 13:04:37.446194 16073 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:37.498340  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 160104us
I20260709 13:04:37.432518 16049 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.506641  8622 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:37.431252 15815 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.451611 15729 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.457816 15995 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.475672 15890 raft_consensus.cc:697] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:37.438413 16050 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:37.521301  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 46405.5us
I20260709 13:04:37.523259  8575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555"
I20260709 13:04:37.526418  8575 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:37.450932 15814 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.547151 16050 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:37.547358 15814 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:37.516287 15303 raft_consensus.cc:2804] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:37.463568 16030 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.529645 15795 consensus_queue.cc:237] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.502302 15914 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 13:04:37.535236 16036 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.536077 16056 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.516592 15926 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:37.519444 15705 consensus_queue.cc:237] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.540437 16042 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:37.570380 16042 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:37.571321 16042 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.537377 15995 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.553907 15839 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:37.510125 16047 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.550251  8575 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 13:04:37.550901  9294 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:37.514853 15882 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:37.524384  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
I20260709 13:04:37.519942 16059 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:37.508706 15997 consensus_queue.cc:1059] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 13:04:37.560739  8584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a0d703ee18464475a290425976246555" is_pre_election: true
I20260709 13:04:37.571835 16084 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.542668  9331 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:37.545452 15985 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.692762  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:37.620994 16097 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:37.693423 16036 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.692541 16049 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.656749 16054 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:37.693487 16072 log.cc:927] Time spent T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
W20260709 13:04:37.693017 16086 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:04:37.698508 16057 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:37.639472 16071 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 13:04:37.564201 16085 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.620715 16035 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.622694 15890 consensus_queue.cc:237] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.780357  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:37.573809 16042 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:37.604358  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45933.7us
I20260709 13:04:37.677327  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:37.791002  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.623274 16060 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.631129 15839 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:37.631410  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.707635 14906 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.621683 15991 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:37.602175  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.796916  8622 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:37.713263  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 162417us
W20260709 13:04:37.723277  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 174879us
I20260709 13:04:37.718226 15995 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:37.742100  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:37.742561 16030 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.743068  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:37.730674 16041 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:37.743945  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:37.733943 16089 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.094s	user 0.006s	sys 0.012s
I20260709 13:04:37.730429 16084 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.731441 16097 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.751763 15538 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.758817  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 3150) took 1217 ms. Trace:
I20260709 13:04:37.834461  8655 rpcz_store.cc:276] 0709 13:04:36.542382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.740770 (+198388us) service_pool.cc:224] Handling call
0709 13:04:37.758430 (+1017660us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b54001d1ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:37.758685 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1274117376}
I20260709 13:04:37.835567  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.835995  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:37.762718 16057 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:37.734411 15303 raft_consensus.cc:697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:37.770263  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.770658  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:37.775882  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:37.776182  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:37.776415  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:37.783701 15792 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.855381  8653 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 23: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:37.793852 14906 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.798619  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.821069  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.821910  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.822908  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.821213  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:37.824204  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.824673  8724 raft_consensus.cc:3060] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:37.825356  8731 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 22.
I20260709 13:04:37.856596  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.863319  8652 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:37.864730  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:37.868340  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.812021 15995 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.783614  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:37.883914  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.827900  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:37.837054 16096 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:37.780375 16085 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.912392  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.913647  8651 raft_consensus.cc:2410] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
W20260709 13:04:37.919556  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 47510.3us
I20260709 13:04:37.920397  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:37.922973  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.940177  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.866411  8650 raft_consensus.cc:2376] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20. Re-sending same reply.
I20260709 13:04:37.960963  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.961620 15814 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:37.962601  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.964017  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.964944  8655 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 22: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:37.851701 16041 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:37.967590 15601 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:37.823515 16101 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:37.968879  8727 raft_consensus.cc:2410] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 13:04:37.969198 16101 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.969537  8728 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:37.969812 16101 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:37.872170 15948 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.973623 16096 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.975111  8659 raft_consensus.cc:2410] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 13:04:37.976071 16096 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.976244 16097 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:37.987391 14906 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:37.999859  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:38.001492  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:38.002564  8656 raft_consensus.cc:2376] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 18. Re-sending same reply.
I20260709 13:04:38.009586  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:38.010530  8654 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:04:38.012360  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:37.972736 15948 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:37.926290 15995 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:38.044917  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:38.037782  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:38.041527  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:37.919742 16049 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:37.834478 15966 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.172s
I20260709 13:04:37.854934 16032 consensus_queue.cc:1059] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.170s
I20260709 13:04:38.097962  8658 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:37.969206 16085 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:38.054145  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:38.055244  8652 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:38.108101  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:38.109304  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:38.109915  8652 raft_consensus.cc:2376] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 18. Re-sending same reply.
I20260709 13:04:38.110922  8652 raft_consensus.cc:1240] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:38.056516  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:37.968798 16041 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:38.070492  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 271348us
I20260709 13:04:38.096757  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:38.101048 16093 raft_consensus.cc:3055] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:38.104516  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:38.118312  8654 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
W20260709 13:04:38.124552  8549 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.853s (SENT)
I20260709 13:04:38.127437  8724 raft_consensus.cc:2468] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:38.051568 16065 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.147903  8656 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:38.163695  9328 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:38.126499 16093 raft_consensus.cc:740] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:38.097805 16049 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:38.172915  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48420.7us
W20260709 13:04:38.174726  8549 leader_election.cc:336] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.921s (SENT)
I20260709 13:04:38.175256  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.138976 16054 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:38.182195  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.185410  8656 raft_consensus.cc:1240] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:38.195868  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 368064us
I20260709 13:04:38.197813  8625 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.202312  8658 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
W20260709 13:04:38.171710  8469 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:38.204831  8625 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.151873 16065 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.179337 15729 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:38.179677 16047 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:38.215518  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 74652.7us
I20260709 13:04:38.217767  8656 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:38.218743  8625 leader_election.cc:400] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20.
I20260709 13:04:38.219316  8625 leader_election.cc:403] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:04:38.213492  8549 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.103s (SENT)
I20260709 13:04:38.220952  8549 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.206825 15705 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:38.230152  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.166272 15303 consensus_queue.cc:237] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:38.234571  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.235271 16035 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20.
I20260709 13:04:38.224905 15839 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:38.235984  8549 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 2.223s (SENT)
W20260709 13:04:38.237672 16102 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 13:04:38.240468 16084 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.243971 16095 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:38.245946 15705 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:38.246465  8625 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.209719 15795 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:38.189666 16054 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.247181 16093 consensus_queue.cc:260] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.227128 15538 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:38.273973 15914 consensus_queue.cc:1059] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:38.278541 16047 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:38.280267 15569 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:38.283135  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.249996 16025 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:04:38.285887  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 159186us
I20260709 13:04:38.286904  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:38.284034 15890 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.033s	sys 0.076s
I20260709 13:04:38.293516  8625 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.285552 16035 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:38.300357  8625 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.290176 16111 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:38.266110 16095 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.293104 15795 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:38.297034 15890 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.034s	sys 0.077s
I20260709 13:04:38.310009  9326 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:38.311604  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 189574us
I20260709 13:04:38.303606 16084 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:38.304405 15839 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:38.318240  9326 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:38.329039 16065 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:38.332623  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.330233 16054 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.306967 16097 consensus_queue.cc:1059] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 13:04:38.337592  8653 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:38.341827  8655 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 13:04:38.347757  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.320659 16049 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:38.364411 16008 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:04:38.388931  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.359998  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.357607 16084 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:38.404554 15995 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.393570 15963 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:38.409312 16084 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:38.416337  9330 raft_consensus.cc:2368] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17.
I20260709 13:04:38.378154 15948 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.393870 15569 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 13:04:38.424466  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.425196  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.426512  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.396663 16054 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.427372  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.428179  8733 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:38.433292  8725 raft_consensus.cc:3060] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:38.428726  8726 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:38.429034  8731 raft_consensus.cc:1240] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: [16.15-16.15]
I20260709 13:04:38.436628  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.398303 16109 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:38.439043  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.420672  8625 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.440368  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.440613  8625 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.441025  8729 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:38.443815 15948 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:38.452888  8549 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Timed out: RequestConsensusVote RPC to 127.7.78.66:39929 timed out after 1.816s (SENT)
I20260709 13:04:38.413976 16085 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:38.454167 16085 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.456519 16085 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:38.457849  8576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34698 (request call id 1424) took 1090 ms. Trace:
I20260709 13:04:38.458446  8576 rpcz_store.cc:276] 0709 13:04:37.367204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.378145 (+ 10941us) service_pool.cc:224] Handling call
0709 13:04:37.401601 (+ 23456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:37.501696 (+100095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:38.457525 (+955829us) spinlock_profiling.cc:243] Waited 77 ms on lock 0x7b5400179b54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:38.457766 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177131264}
W20260709 13:04:38.428297 16107 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.185s	user 0.001s	sys 0.006s
I20260709 13:04:38.421069 16084 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.463068 16095 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:38.463770 16111 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:38.464087 16084 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:38.477063  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.478153  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.414171 16041 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:38.485862  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.486240  8729 raft_consensus.cc:1240] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:38.488624  8622 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.488973  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:38.489298  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.489001  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:38.432583 15889 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:38.491353  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.491740  8549 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.494362  8724 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:04:38.490119  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.438535 15995 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:38.492818  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.494982  8726 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:38.495358  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.414330 15963 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:38.495676  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:38.485749 16101 consensus_meta.cc:287] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.515s	user 0.013s	sys 0.000s
I20260709 13:04:38.494030 16041 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.498260  8731 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 22.
I20260709 13:04:38.500015  8725 raft_consensus.cc:2468] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:38.501125  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.518260  8729 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:38.518354  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.499483  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.489584 15504 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:38.514390  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.516055 14906 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.522601  8731 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:38.522980 14906 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.516427  8732 raft_consensus.cc:1275] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:38.521546  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.526320  8730 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 14: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:38.526221  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.438771 16054 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.519603 15814 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:38.528442  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.527292  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:38.531826  9294 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:38.536315  8622 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:38.508518 16008 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:38.535228  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.536779  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:38.538434  8622 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:38.539026  8622 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.542004  8625 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.542897  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.543565  8656 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 13:04:38.538115  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:38.545607  9294 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:38.546293  9294 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:38.546926  9294 leader_election.cc:336] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:38.547706 16097 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:38.548024 16093 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:38.542409  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.551605 16035 consensus_queue.cc:1059] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:04:38.552261  8657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1228) took 2576 ms (client timeout 1918 ms). Trace:
I20260709 13:04:38.552937 16097 raft_consensus.cc:3055] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:38.553383  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:38.553385 16097 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:38.554616 16097 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.556025 15963 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:38.557816  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:38.558827 15992 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:38.559227 15992 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:38.559588 15992 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:38.567209 14906 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:38.552695  8657 rpcz_store.cc:269] 0709 13:04:35.975398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.975522 (+   124us) service_pool.cc:224] Handling call
0709 13:04:38.552049 (+2576527us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400352ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:38.552196 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3088247040}
W20260709 13:04:38.539522  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.570863  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:38.571553  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:38.516892 16101 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.512317 15889 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:38.574801  8622 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:38.567255 15855 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:38.533594 16121 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:38.567476 15504 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.578869  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.573451 15569 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:38.583076 16101 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:38.584949 15889 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.586671 16054 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.566684 16008 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:38.576231 16100 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.619112 16100 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.621124  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:38.619159 16008 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.569949 15948 consensus_queue.cc:1048] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 13:04:38.563423 15814 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.628273  8730 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:38.629455  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.616264  9327 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1576) took 1199 ms. Trace:
I20260709 13:04:38.633642  9327 rpcz_store.cc:276] 0709 13:04:37.416372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.422421 (+  6049us) service_pool.cc:224] Handling call
0709 13:04:37.423065 (+   644us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:37.549096 (+126031us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.549128 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:37.549188 (+    60us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:37.549222 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:38.616021 (+1066799us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54007dcdd4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:38.616073 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:38.616205 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289453952}
I20260709 13:04:38.635603  8733 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:04:38.637034  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:38.637779  8656 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
I20260709 13:04:38.604562  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:38.638747  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.574482 16059 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:38.616359  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.598443  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.644166  8725 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:38.638797  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 64027.1us
I20260709 13:04:38.639700  8732 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:38.593693  8550 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.78.67:37975 timed out after 1.692s (SENT) [suppressed 60 similar messages]
I20260709 13:04:38.646209  9327 raft_consensus.cc:1275] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 13:04:38.596844  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.576906 16083 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:38.589301 16042 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.781s	user 0.001s	sys 0.008s
I20260709 13:04:38.639493  8657 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
W20260709 13:04:38.647507  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 53862us
I20260709 13:04:38.658989  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:38.654175  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 57479.2us
I20260709 13:04:38.655794  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:38.660786  8726 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:38.658236  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:38.675101  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 58857.8us
I20260709 13:04:38.675884  9323 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.675992  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.682037 15569 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:38.674394 16049 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:38.684190 16053 consensus_queue.cc:1059] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:38.685492 16053 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:38.678026 15705 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.648346 15948 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:38.674178  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:38.694300  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:38.694919  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:38.696921  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.657861 16054 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:38.698180  8726 raft_consensus.cc:1240] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:38.697115  9329 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
W20260709 13:04:38.705312  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:38.710857 16113 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.457s	user 0.000s	sys 0.003s
W20260709 13:04:38.714007 16106 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.473s	user 0.004s	sys 0.000s
I20260709 13:04:38.714428  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:38.714442 15729 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:38.715477 15995 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.639164 15504 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:38.718662  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.727474  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.696197 16042 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:38.674624 16047 consensus_queue.cc:1059] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 13:04:38.671908 15992 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.733834  8724 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:38.703905 16059 raft_consensus.cc:3055] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:38.672312 16083 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.675384 16121 consensus_queue.cc:1059] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 13:04:38.741945  9323 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:38.743553  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.735512  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.754142  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43648 (request call id 1593) took 1343 ms. Trace:
I20260709 13:04:38.730937 15538 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:38.759305  9324 rpcz_store.cc:276] 0709 13:04:37.410576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.411496 (+   920us) service_pool.cc:224] Handling call
0709 13:04:37.411847 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:37.418117 (+  6270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.418156 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:37.434331 (+ 16175us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.767713 (+333382us) spinlock_profiling.cc:243] Waited 8.2 ms on lock 0x7b18002edbc0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b28c02 00007f14097b020b 00007f14097b5d23 00007f14097b60d1 00007f1409bd1a0f
0709 13:04:37.977026 (+209313us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:37.977065 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.977138 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:38.713328 (+736190us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:38.753896 (+ 40568us) raft_consensus.cc:1685] finished
0709 13:04:38.753917 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:38.754068 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1429,"raft.queue_time_us":39134,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":11106,"spinlock_wait_cycles":57116160,"thread_start_us":209119,"threads_started":1,"wal-append.queue_time_us":741616}
I20260709 13:04:38.695502 15705 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.755666  8725 raft_consensus.cc:2468] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 23.
I20260709 13:04:38.760574  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:38.745456 16065 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.744249 15995 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.746271 15889 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.772612  8733 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 16. (term mismatch)
I20260709 13:04:38.701301 16053 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.789812  8652 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 20. Ops: []
I20260709 13:04:38.747368 15948 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.765793 16101 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.787432  8733 pending_rounds.cc:85] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting all ops after (but not including) 15
I20260709 13:04:38.793073  8733 pending_rounds.cc:107] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Aborting uncommitted NO_OP operation due to leader change: 18.16
I20260709 13:04:38.793483  8733 raft_consensus.cc:2889] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:38.756110 16059 raft_consensus.cc:740] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:38.788278  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 3120) took 1012 ms. Trace:
I20260709 13:04:38.773798 16083 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.770890  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.794389  8728 rpcz_store.cc:276] 0709 13:04:37.775758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.775896 (+   138us) service_pool.cc:224] Handling call
0709 13:04:38.788062 (+1012166us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54004727d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:38.788209 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":262762368}
I20260709 13:04:38.761487 16047 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:04:38.795428  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 116922us
I20260709 13:04:38.766558 15538 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:38.786531 16049 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:38.806591  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 65494.5us
I20260709 13:04:38.782034 15729 consensus_queue.cc:1059] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 13:04:38.805326 15538 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.792555 15889 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.849931  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1581) took 1423 ms. Trace:
I20260709 13:04:38.796262 16101 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.858839  9331 rpcz_store.cc:276] 0709 13:04:37.426162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.426287 (+   125us) service_pool.cc:224] Handling call
0709 13:04:38.849758 (+1423471us) spinlock_profiling.cc:243] Waited 98.8 ms on lock 0x7b5400551c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:38.849859 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227216256}
I20260709 13:04:38.854058  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:38.815990 15666 raft_consensus.cc:3055] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:38.861490  8727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40420 (request call id 3119) took 1086 ms. Trace:
I20260709 13:04:38.820257 16047 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:38.865197  8727 rpcz_store.cc:276] 0709 13:04:37.775440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.775588 (+   148us) service_pool.cc:224] Handling call
0709 13:04:38.849452 (+1073864us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400203ad4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:38.861411 (+ 11959us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261729664}
I20260709 13:04:38.867481  8728 raft_consensus.cc:2393] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:38.795871 15948 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.866138 15666 raft_consensus.cc:740] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:38.897898  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 155214us
I20260709 13:04:38.903216  9296 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:38.870963 16128 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:38.913125  8731 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
W20260709 13:04:38.915596  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 171310us
I20260709 13:04:38.916441  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:38.866353 15890 consensus_queue.cc:1059] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 13:04:38.891856 16126 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:38.858289 16059 consensus_queue.cc:260] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.916324 15504 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:38.957130  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 156204us
I20260709 13:04:38.955063 15992 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.920099 16128 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.982569 16041 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:38.986423  9323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1587) took 1363 ms (client timeout 1723 ms). Trace:
I20260709 13:04:38.986617 15948 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.975463 15666 consensus_queue.cc:260] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:38.988502 15666 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:38.991048 16128 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:38.991734 15839 raft_consensus.cc:3055] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:38.992432 15839 raft_consensus.cc:740] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:38.995038 15985 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.995632 15985 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:38.993312 15839 consensus_queue.cc:260] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.006078 15889 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:38.999486 16137 raft_consensus.cc:3055] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:39.007241 16137 raft_consensus.cc:740] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:39.007445 15980 raft_consensus.cc:3055] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:39.007895 15980 raft_consensus.cc:740] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:38.986991  9323 rpcz_store.cc:269] 0709 13:04:37.623536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.770414 (+146878us) service_pool.cc:224] Handling call
0709 13:04:38.986201 (+1215787us) spinlock_profiling.cc:243] Waited 822 ms on lock 0x7b540037a5d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:38.986346 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1889466112}
I20260709 13:04:39.008265 16137 consensus_queue.cc:260] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.008754 15980 consensus_queue.cc:260] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.009907 15980 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.967437 15504 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 13:04:39.011658 16053 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:38.961757 16126 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 13:04:39.015075  8652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1254) took 1225 ms. Trace:
I20260709 13:04:39.015514  8652 rpcz_store.cc:276] 0709 13:04:37.789680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.124806 (+335126us) service_pool.cc:224] Handling call
0709 13:04:38.789681 (+664875us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:38.790202 (+   521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.014867 (+224665us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b540031c750. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:39.015011 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1528173312}
I20260709 13:04:39.016801 15985 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.019943 16049 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:39.020891 16049 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:39.020970 16137 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:39.005136 16101 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.020107  8653 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:39.000478 16100 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.004992 16042 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:39.047765  8623 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:39.051544  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 135179us
I20260709 13:04:39.052820  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:38.977111 16059 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:39.059650  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:39.047444 16049 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:39.063740  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.086293  8652 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 23: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:39.047008 16093 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.087330 16093 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:39.067761  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 99799.7us
I20260709 13:04:39.090659  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:39.091884 16049 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:39.092320  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:39.054488 16101 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:39.092371 16049 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:39.093973 16049 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:39.053634 15991 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:39.097649  8654 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:39.093144  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:39.074944 16139 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.004s
I20260709 13:04:39.069900 16065 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:39.057915 15855 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:39.100461 15855 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.101884 16127 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:39.103039 15855 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.103374 16127 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:39.103771 15855 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:39.097810 15991 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.107234 15991 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.105304 15855 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:39.071475  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:39.109908 16093 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.111220 16065 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:39.113197  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:39.114895 16025 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:39.117105  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:39.116025  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.123299  8731 raft_consensus.cc:2376] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 16. Re-sending same reply.
I20260709 13:04:39.124820 16065 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:39.117901  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.127717  8732 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:39.119252  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.128983  8724 raft_consensus.cc:2468] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 22.
I20260709 13:04:39.130163 16065 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:39.132691  9297 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:39.134634 15839 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:39.140018 16065 raft_consensus.cc:2804] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:39.140352 15995 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.140681 16065 raft_consensus.cc:697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:39.122411 16025 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:39.115444  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.150033  8550 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:39.141748 16065 consensus_queue.cc:237] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.151067 15666 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:39.151657 15666 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:39.152520 15666 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.154095 16097 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:39.155851 15705 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.157352 16056 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.158447  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.159988  8724 raft_consensus.cc:1275] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:39.161725 16025 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.169883  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:39.170616  8724 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:39.173705  8731 raft_consensus.cc:1240] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:39.173628  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:39.176409  8654 raft_consensus.cc:2468] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
W20260709 13:04:39.177706  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 274588us
I20260709 13:04:39.175289  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.183928  9296 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:39.184031  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.187840  8659 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:39.189059  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.192150  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:39.192178  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.196087  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.210569  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.211375  8731 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:39.212417  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.213117  8731 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:39.218501  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:39.219766  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.219715  8731 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:39.221887 15985 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:39.222469 15985 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.224468  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.225850  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:39.226646  8732 raft_consensus.cc:2393] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:39.226547  8625 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:39.227728  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:39.228307 16111 consensus_queue.cc:1059] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:39.229650 16111 raft_consensus.cc:2804] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:39.230043 16093 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:39.230247 16111 raft_consensus.cc:697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:39.230135  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56593.2us
I20260709 13:04:39.230902 16020 raft_consensus.cc:3055] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:39.224794 15985 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.231380 16020 raft_consensus.cc:740] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:39.231324  8550 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:39.232831 15814 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:39.230468 15855 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:39.233880 15855 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:39.234277 15855 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:39.235199  8469 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:39.236119  8726 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:39.237887 15985 raft_consensus.cc:2804] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:39.238353 15985 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:39.238756 15985 raft_consensus.cc:3060] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:39.229012  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:39.120947 16049 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.232584 16020 consensus_queue.cc:260] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.188465 16121 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:39.260609  9330 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1586) took 1710 ms (client timeout 1759 ms). Trace:
I20260709 13:04:39.296178  8625 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:39.306521  9330 rpcz_store.cc:269] 0709 13:04:37.550333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.692451 (+142118us) service_pool.cc:224] Handling call
0709 13:04:39.260406 (+1567955us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b54002779d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:39.260549 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1602717056}
I20260709 13:04:39.241865  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:39.308068  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66342.4us
W20260709 13:04:39.308511  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56855.2us
W20260709 13:04:39.262501 15914 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.857s	user 0.002s	sys 0.014s
W20260709 13:04:39.272045  8729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.011s	sys 0.007s
W20260709 13:04:39.312045  8729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.012s	sys 0.008s
W20260709 13:04:39.293895  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82765.7us
I20260709 13:04:39.222375 16147 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 13:04:39.303027  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.295856  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:39.314653  9294 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:39.319653  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:39.317746  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.320640  8653 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:39.322621  8724 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:39.334484  8732 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:39.347690  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51571.6us
W20260709 13:04:39.311987 15914 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.907s	user 0.002s	sys 0.014s
I20260709 13:04:39.230618 16093 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:39.348716  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:39.350263  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.294518 15839 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:39.305927 16025 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.376631 16093 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:39.312773 15995 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:39.383827  8731 raft_consensus.cc:2468] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:39.385684  8625 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:39.342664 16053 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:39.389429  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 80040.5us
I20260709 13:04:39.343245 15815 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:39.353170 15504 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:39.394650  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.394115  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.402772  8652 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:39.373798 16020 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:39.396682  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.397996  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.395898  8731 raft_consensus.cc:1275] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:04:39.313176 16101 consensus_queue.cc:1048] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
W20260709 13:04:39.402925  8624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.400436  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.412416  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.413100  8654 raft_consensus.cc:2468] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:39.231297 16111 consensus_queue.cc:237] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.371822 16139 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 13:04:39.317082 16121 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.467002  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:39.398875  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 85097.2us
I20260709 13:04:39.469092  8731 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:39.433112  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.479914 15814 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.423996 15995 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:39.479136  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 170757us
I20260709 13:04:39.390061 16025 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.391673 16147 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.137s
I20260709 13:04:39.480083 16056 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.532215 16056 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.536648 16109 consensus_queue.cc:1048] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:39.537788 16142 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:39.538269 16142 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:39.538611 16142 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:39.541277  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:39.424846 15815 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.529619 15814 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.488374  8650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3165) took 1685 ms. Trace:
I20260709 13:04:39.484318  8698 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:39.598590  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 57401.4us
I20260709 13:04:39.399788  9297 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:39.583362 16049 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:39.594798  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 68598.9us
I20260709 13:04:39.532265  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:39.606304  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 206576us
W20260709 13:04:39.607331  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 123138us
I20260709 13:04:39.607972  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:39.604359  8650 rpcz_store.cc:276] 0709 13:04:37.802854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.134378 (+331524us) service_pool.cc:224] Handling call
0709 13:04:38.714520 (+580142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:38.714718 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:38.863719 (+149001us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.067463 (+203744us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.067525 (+    62us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:39.067576 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.067643 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.488099 (+420456us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b540018fbd0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:39.488284 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":195178,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":3728,"raft.queue_time_us":462,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":28519,"spinlock_wait_cycles":1280470400,"thread_start_us":346999,"threads_started":2,"wal-append.queue_time_us":461002}
I20260709 13:04:39.391763 16016 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:39.677222  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:39.501196 15538 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.641s	user 0.013s	sys 0.000s
I20260709 13:04:39.397560 15792 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:04:39.633075  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92602us
I20260709 13:04:39.596357 15705 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:39.585381 15995 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:39.667646  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:39.394958 16151 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.242s
I20260709 13:04:39.455168 15569 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:39.669948  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 62057.6us
I20260709 13:04:39.713117 15569 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.620391 16147 raft_consensus.cc:2804] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:39.685910  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:39.630808 16164 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.635681 16041 raft_consensus.cc:2804] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:39.732025  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:39.455901 15504 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.401002 16053 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.711320 16056 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.677852 16016 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:39.711135 15995 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:39.711817  8622 leader_election.cc:336] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:39.696229 15538 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.643208 16020 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.779084 15995 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:39.779237 16020 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.779749 15995 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:39.780259 15995 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.711164  9324 raft_consensus.cc:1275] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 13:04:39.745702 16131 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:04:39.738121  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.620146 16097 consensus_queue.cc:1048] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 13:04:39.741190  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:39.732820  9328 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 23.
I20260709 13:04:39.745328  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.788795  8650 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:39.682693 16129 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:39.772547 16016 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:39.768543 15666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.053s	sys 0.072s
I20260709 13:04:39.773521  8622 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:39.711400 15792 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:39.783004  9295 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.687907  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:39.795300  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 83530.2us
I20260709 13:04:39.795665  9323 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:39.757009 16161 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
W20260709 13:04:39.770835 16162 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
W20260709 13:04:39.773006 16150 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.375s	user 0.001s	sys 0.000s
I20260709 13:04:39.728084  8731 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:39.839507  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.769143 16041 raft_consensus.cc:697] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:39.788007  9325 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 22.
W20260709 13:04:39.784641 16144 log.cc:927] Time spent T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.397s	user 0.001s	sys 0.001s
I20260709 13:04:39.696278 16165 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 13:04:39.843669  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47527.2us
W20260709 13:04:39.794384 15666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.053s	sys 0.072s
I20260709 13:04:39.745640 16164 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:39.857961  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:39.861202  8696 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:39.838117 16140 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:39.859706  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:39.861308  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 148270us
I20260709 13:04:39.861704  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:39.861361  9323 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
W20260709 13:04:39.866674  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 490265us
W20260709 13:04:39.867226  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 181395us
W20260709 13:04:39.869256  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:39.870513  8696 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:39.871450  8696 leader_election.cc:336] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:39.862344  8650 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:39.872826  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.874388  8650 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
W20260709 13:04:39.856222 16149 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.370s	user 0.001s	sys 0.000s
I20260709 13:04:39.875173  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:39.876408  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:39.874146  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:39.882722  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:39.868438 16165 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:04:39.881644 15855 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.647s	user 0.007s	sys 0.010s
I20260709 13:04:39.870280 16147 raft_consensus.cc:697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:39.887184 16085 raft_consensus.cc:3055] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:39.895200 15814 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:39.896047 16166 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:39.896884  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3273) took 1255 ms. Trace:
I20260709 13:04:39.879969 16095 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:39.901965 16164 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:39.902863 16156 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.882264 16020 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.887758 16121 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.907601 16134 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:04:39.907768 16154 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:04:39.909314 16041 consensus_queue.cc:237] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.913121 16165 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:39.920713 15985 consensus_meta.cc:287] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.001s
I20260709 13:04:39.921113 15538 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.922154 15504 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:39.922935 16153 log.cc:927] Time spent T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 13:04:39.923461 16025 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.887292 15569 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:39.926766 16176 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:39.864902 15815 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:39.878046 16148 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.000s	sys 0.009s
I20260709 13:04:39.930859  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:39.933959  8656 raft_consensus.cc:1275] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 13:04:39.943945  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63338.7us
I20260709 13:04:39.943742  8730 rpcz_store.cc:276] 0709 13:04:38.641838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.641954 (+   116us) service_pool.cc:224] Handling call
0709 13:04:38.849593 (+207639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:38.849820 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:39.310870 (+461050us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.459870 (+149000us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.459946 (+    76us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:39.460014 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.460104 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.896006 (+435902us) spinlock_profiling.cc:243] Waited 3.27 ms on lock 0x7b18004c1f20. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4
0709 13:04:39.896708 (+   702us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400203ad0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:39.896822 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":36664,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":713,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":10474,"spinlock_wait_cycles":492882944,"thread_start_us":581899,"threads_started":2,"wal-append.queue_time_us":433777}
I20260709 13:04:39.940263  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:04:39.932531 16127 consensus_meta.cc:287] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.829s	user 0.010s	sys 0.002s
I20260709 13:04:40.067270  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.067617  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:40.067962  9327 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:39.931938 16085 raft_consensus.cc:740] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:39.951195 16053 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:39.975039  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 45665.9us
I20260709 13:04:39.950942 16134 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:39.888926 15792 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:39.963457 16054 raft_consensus.cc:3055] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:40.079033  8730 raft_consensus.cc:1275] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:39.952741 16165 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:40.081538  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:39.951176 16164 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:39.932744 15855 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:39.977286 16148 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.000s	sys 0.010s
I20260709 13:04:40.062698  9323 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:40.064061  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:39.966378 16156 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:39.956609 15985 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.074281  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:40.099561  9325 raft_consensus.cc:2410] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 17 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20260709 13:04:40.076261 16134 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:40.074869  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:40.052726 15705 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:40.078482 16054 raft_consensus.cc:740] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:39.951424 16166 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.137768 16049 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:40.138255 16049 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:40.139391 16166 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:40.139150 16049 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.140478 16049 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:40.090422  9326 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:40.080873  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:40.069234 15815 raft_consensus.cc:2804] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:40.056896 16100 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:40.093353  9331 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:40.073045 16127 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:40.094859  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.070259 15995 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:39.972803 16176 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:40.065078  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 124935us
I20260709 13:04:40.182281  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:40.097856  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.182891  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:40.106559 16155 log.cc:927] Time spent T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
I20260709 13:04:40.107589  9325 raft_consensus.cc:1240] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 22. Current term is 23. Ops: []
W20260709 13:04:40.110023 16158 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
W20260709 13:04:40.114085 16170 log.cc:927] Time spent T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:40.188814  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.189595  9331 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:40.114890 16156 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:40.123875 16142 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.015s
I20260709 13:04:40.127588 16054 consensus_queue.cc:260] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:40.128851 16172 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 13:04:40.129204 15705 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.104187 16042 consensus_queue.cc:1048] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
W20260709 13:04:40.153142  8625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:40.234867 15991 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.235414 15991 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.236768  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.162480 16085 consensus_queue.cc:260] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.237519 15991 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.237805 16085 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:40.193078 16111 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:40.243049 16111 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.166931 15855 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:40.169661  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 88859.4us
I20260709 13:04:40.261926  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:40.180081  9329 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:40.178826 16100 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:40.183859  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:40.188058 15980 consensus_queue.cc:1048] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 13:04:40.179000 15815 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.192164  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:40.182416 16129 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:40.179513 16095 raft_consensus.cc:2764] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:04:40.209908  9325 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:04:40.210054  9330 raft_consensus.cc:1240] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:40.211103  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.211848  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.212514 15995 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.229573 15985 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.232014 16176 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.232580 16025 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:40.246285  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:40.246692 16159 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.186s	user 0.002s	sys 0.000s
I20260709 13:04:40.055630 16147 consensus_queue.cc:237] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.255283 15705 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.255514  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.265082  9331 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:40.263381  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:40.295205  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:40.196020 16093 consensus_meta.cc:287] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.800s	user 0.017s	sys 0.001s
I20260709 13:04:40.294662  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:40.264994 16111 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:40.272993 15890 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:40.308259  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:40.309803  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3283) took 1193 ms. Trace:
I20260709 13:04:40.313442  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.265791 16100 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:40.277175  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 93381us
W20260709 13:04:40.316830  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53516.8us
I20260709 13:04:40.265875 16042 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:40.304049 15815 raft_consensus.cc:3060] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:40.321516  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:40.327637  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:40.352995  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:40.266165 16053 consensus_queue.cc:1175] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d0a2b9c938d94a4fa258e068ea054d84" responder_term: 21 status { last_received { term: 21 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 21 index: 15 } }
I20260709 13:04:40.355707  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.356261  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.355844  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.328027  8651 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:40.259663 16142 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.305701 16056 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:40.326468  8725 rpcz_store.cc:276] 0709 13:04:39.116349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.120072 (+  3723us) service_pool.cc:224] Handling call
0709 13:04:39.120469 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:39.120923 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:39.120967 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:39.121612 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.271163 (+149551us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:39.271218 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.271326 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:40.309556 (+1038230us) raft_consensus.cc:1685] finished
0709 13:04:40.309584 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:40.309735 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4902,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":86,"thread_start_us":149052,"threads_started":1,"wal-append.queue_time_us":756279}
I20260709 13:04:40.359354  9328 raft_consensus.cc:2468] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
W20260709 13:04:40.321316 16172 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:04:40.273547 16054 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:40.314910 16020 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:04:40.381076  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:40.309803 16025 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:40.331319  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:40.402328  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 71109us
I20260709 13:04:40.403478  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:40.403755  8622 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:40.404559  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:40.352542 15991 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:40.409904  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:40.410928  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:40.428443  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:40.411723  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.348747 15890 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.420222  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:40.426424  8653 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:40.436391 16184 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
W20260709 13:04:40.437330 16167 log.cc:927] Time spent T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.591s	user 0.002s	sys 0.000s
I20260709 13:04:40.315274 15995 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.399963 15569 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.306420 16129 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.396080 16166 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:40.361299 15814 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:40.458276  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.401185 16020 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.459352  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.407349 16025 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:40.399364 16121 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:40.426944 16097 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:40.353533 16042 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:40.468839  8696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:40.470630  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:40.350071 16093 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.473058  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.427416 15504 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:40.470208 16182 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:40.507678  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.362149 15985 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.103s
W20260709 13:04:40.477968  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 68825.9us
I20260709 13:04:40.478334  8728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3284) took 1356 ms. Trace:
I20260709 13:04:40.486922  9322 raft_consensus.cc:2410] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 17 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20260709 13:04:40.489001  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:40.533386 16187 log.cc:927] Time spent T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 13:04:40.492343  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.477918 15666 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.475212 15814 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:40.432132 16056 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.509347  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:40.573664  8656 raft_consensus.cc:1240] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:40.565920  9328 raft_consensus.cc:2376] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a0d703ee18464475a290425976246555 in term 20. Re-sending same reply.
I20260709 13:04:40.490690 16097 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.474251 15569 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.495129 16121 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.484560 16020 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:40.479163 16047 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:40.483143 15855 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.593575  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.471592 16152 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.098s
I20260709 13:04:40.457516 15890 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:40.597270  8728 rpcz_store.cc:276] 0709 13:04:39.122433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.123745 (+  1312us) service_pool.cc:224] Handling call
0709 13:04:39.124026 (+   281us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:39.124355 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:39.124379 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:39.124820 (+   441us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.295124 (+170304us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:39.295752 (+   628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.295914 (+   162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:40.245959 (+950045us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:40.461149 (+215190us) raft_consensus.cc:1685] finished
0709 13:04:40.461172 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:40.478254 (+ 17082us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1683,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":60,"spinlock_wait_cycles":13901568,"thread_start_us":163135,"threads_started":1,"wal-append.queue_time_us":831963}
I20260709 13:04:40.600579  9330 raft_consensus.cc:2468] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:40.533607 16182 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.601696  8732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 3142) took 1010 ms. Trace:
I20260709 13:04:40.533533 16145 raft_consensus.cc:2804] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:40.622185  8732 rpcz_store.cc:276] 0709 13:04:39.591489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.591707 (+   218us) service_pool.cc:224] Handling call
0709 13:04:39.592146 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:39.641789 (+ 49643us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:39.641830 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:39.900405 (+258575us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b2c000b2cd8. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140850fa11 00007f140851f372 00007f1409c51a01 00007f1409c1643a 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2
0709 13:04:39.900872 (+   467us) log.cc:844] Serialized 26 byte log entry
0709 13:04:40.078144 (+177272us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:40.078181 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:40.078246 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:40.598735 (+520489us) raft_consensus.cc:1685] finished
0709 13:04:40.598756 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:40.601629 (+  2873us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16013,"raft.queue_time_us":92936,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":57855744,"thread_start_us":194849,"threads_started":2,"wal-append.queue_time_us":418201}
I20260709 13:04:40.603371  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.606128 15814 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:40.609015  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:40.609786 16097 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:40.605852 15666 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.615749  8657 raft_consensus.cc:2393] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 21: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
W20260709 13:04:40.606940 16016 consensus_meta.cc:287] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.767s	user 0.008s	sys 0.004s
I20260709 13:04:40.622881  9330 raft_consensus.cc:1240] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 14. Current term is 15. Ops: []
W20260709 13:04:40.647886  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 244559us
I20260709 13:04:40.568706 15795 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.656966  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.595337 15504 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.605293 16142 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.648643  9330 raft_consensus.cc:1240] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:40.623431  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:40.631876  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 298272us
I20260709 13:04:40.632261 16182 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:40.661803  9326 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 23: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:40.663894  8657 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:40.604504 16165 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.632119 16145 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:40.668834  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:40.670786  8469 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:40.673048  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:40.673388  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:40.673743  8657 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:40.674616  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:40.684998  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.685717  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:40.686408  8657 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
W20260709 13:04:40.665174 16049 consensus_meta.cc:287] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.524s	user 0.009s	sys 0.004s
I20260709 13:04:40.697268  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:40.697518  9324 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:40.661094 15795 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.632380 15855 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.703685  9329 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:40.676262 16145 raft_consensus.cc:3060] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:40.713078  9294 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.005s	sys 0.009s
W20260709 13:04:40.719766  9294 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.005s	sys 0.009s
W20260709 13:04:40.720175  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 547027us
I20260709 13:04:40.676461 16142 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:40.684468 15985 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.676553 16093 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.653313 16016 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.736266  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.744949  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.739146 16189 consensus_queue.cc:1048] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 13:04:40.754330  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:40.721524 16156 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.727588 16152 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:40.761587 15995 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.772184  8659 raft_consensus.cc:2368] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19.
I20260709 13:04:40.781980  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:40.755900 16109 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:40.755627 16176 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:04:40.795513  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 98323.5us
I20260709 13:04:40.735699 16142 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.811825  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38088 (request call id 3180) took 1406 ms. Trace:
I20260709 13:04:40.774210 16129 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.813977 16202 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:40.815033 15795 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.815294 15666 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.815641 15795 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:40.816174 15795 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.820983  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:40.822285 16165 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.824425 16188 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:40.827476 15855 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.828236 15855 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:40.822309 16197 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:40.830134 16197 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.830487 16197 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:40.830973 15795 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.832078 16016 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.834035 15504 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.834729 16121 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.837603 15569 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.838964 16056 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.840438 15815 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:40.788549 16047 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
W20260709 13:04:40.843055 16085 consensus_meta.cc:287] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.605s	user 0.012s	sys 0.000s
I20260709 13:04:40.844030  8730 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 23.
I20260709 13:04:40.785463 16152 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.847625 16152 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.799018 16176 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.851341 16176 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:40.812397  8653 rpcz_store.cc:276] 0709 13:04:39.396157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.396382 (+   225us) service_pool.cc:224] Handling call
0709 13:04:40.801096 (+1404714us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400470754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:40.801274 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2351415552}
I20260709 13:04:40.859975  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:40.861756  8654 raft_consensus.cc:2368] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20.
I20260709 13:04:40.796854  8548 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:40.771413 16126 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:40.773465 16134 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.604s	user 0.013s	sys 0.002s
I20260709 13:04:40.771390 16042 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:40.744824 15985 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:40.868129  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 71357.5us
I20260709 13:04:40.782160 16156 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:40.870988  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:40.879428  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:40.828823 15855 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.825093 16188 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:40.895658  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 106270us
I20260709 13:04:40.878891 15991 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:40.867298 16129 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:40.895385  8548 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:40.802358 16109 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.896911 15569 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:40.894562 16042 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:40.922008  8696 leader_election.cc:336] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:40.878221 16134 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.938383  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:40.929688  8696 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:40.952821  8696 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:40.953737  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.936749  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.957576  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.958072  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.958392  8733 raft_consensus.cc:2393] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 23: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:40.958920  8724 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:40.959380 16020 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.960045  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:40.960682  8729 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:40.962393 16142 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.963981 15985 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.966295  8652 raft_consensus.cc:1275] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:40.968384  8729 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
W20260709 13:04:40.968570  8696 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:40.970968  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:40.972029  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.969106  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:40.974064  8731 raft_consensus.cc:2393] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 18: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:40.951035 16042 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.982095 15815 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.934106 15569 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:40.983820  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:40.983736 15569 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.990880 16188 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:40.992185 16205 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:04:40.925009 16189 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:40.992944 16156 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:40.986826 16109 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:40.993095 16189 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:40.993837 16207 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:04:40.995608 16204 log.cc:927] Time spent T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 13:04:41.085757  8655 raft_consensus.cc:1240] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:40.997494  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.933252 15991 raft_consensus.cc:2804] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:41.001593 16152 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:41.103164  9323 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:41.107621  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:41.002920 16111 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.643s	user 0.010s	sys 0.001s
I20260709 13:04:40.982805  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.969667  8724 raft_consensus.cc:1275] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
W20260709 13:04:41.085610  8659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38104 (request call id 3593) took 1710 ms (client timeout 2004 ms). Trace:
W20260709 13:04:40.970861  8696 leader_election.cc:336] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.120275  8650 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 23: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:41.125579  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 171368us
I20260709 13:04:40.913899 16129 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.131810  8697 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:41.127592  8659 rpcz_store.cc:269] 0709 13:04:39.374915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.375107 (+   192us) service_pool.cc:224] Handling call
0709 13:04:41.045613 (+1670506us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54001310d4. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:41.085548 (+ 39935us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3148898176}
W20260709 13:04:41.009518 16100 consensus_meta.cc:287] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.648s	user 0.011s	sys 0.000s
I20260709 13:04:41.008952 15985 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.131469  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.146538  8650 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:41.147476  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.135262  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:40.981288 16042 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:41.155472  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 44655.3us
I20260709 13:04:41.156407  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:41.128175  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 157333us
I20260709 13:04:41.137710  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:41.033792 16188 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:41.166429  8696 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.166955  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.166764 16188 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.151355  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.168259  8729 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 14.
I20260709 13:04:41.168478  8656 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:40.991451 15890 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.104103 15991 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:41.130460 15980 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:41.160713  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.170351  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:41.172923  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.178172  8659 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:41.135089 16053 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.136695 15795 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:41.079567 16210 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:41.151485 16207 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:41.134846 16093 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:41.199623  8724 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:41.143424 16142 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.184021  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.148470 16020 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.136309 16059 consensus_queue.cc:1059] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.054s
W20260709 13:04:41.195143  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 56251.2us
I20260709 13:04:41.213881 16127 raft_consensus.cc:3055] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:41.214362 16127 raft_consensus.cc:740] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:41.215842 15855 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.216403 15855 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.216848 15855 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.219033 15855 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.191517 16213 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:41.176148 15991 raft_consensus.cc:3060] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:41.221834 16175 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:41.222465 16129 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.223084  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.224257  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:41.225560  9294 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:41.225854 16097 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.000s
I20260709 13:04:41.226037  9294 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:41.226403 16097 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.226891 16178 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:41.227511 15792 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:41.227907 15792 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:41.228281 15792 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:41.230206 16189 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.190615 16053 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:41.231249  9294 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.231735  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.232477 16178 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:41.233057 16053 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:41.233799  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:41.254571  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:41.255177  9294 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.255542  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.255687  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:41.256843  9294 leader_election.cc:336] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:41.234418 16182 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.550s	user 0.009s	sys 0.001s
I20260709 13:04:41.135366 16056 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 13:04:41.192044 15980 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.274489  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.275226  9325 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:41.276194 15980 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:41.277062  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:41.277333 15814 consensus_meta.cc:287] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.629s	user 0.006s	sys 0.005s
I20260709 13:04:41.277817 15814 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.278308  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:41.187737  8730 raft_consensus.cc:2410] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:41.279246  8728 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:41.225245  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:41.258263  9294 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.262316  9330 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:41.139310 16111 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.258579  8698 leader_election.cc:400] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20.
I20260709 13:04:41.286291  8698 leader_election.cc:403] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:04:41.236344 16025 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.741s	user 0.008s	sys 0.001s
I20260709 13:04:41.217085 16176 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.302078  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.302999  9330 raft_consensus.cc:2393] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 20: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:41.254153 16142 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.304047  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:41.304970 16142 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.305263  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:41.284938 16182 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.306211  9324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1368) took 1243 ms. Trace:
I20260709 13:04:41.306409  9331 raft_consensus.cc:2368] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19.
I20260709 13:04:41.306617  9324 rpcz_store.cc:276] 0709 13:04:40.063228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.063391 (+   163us) service_pool.cc:224] Handling call
0709 13:04:40.063729 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:40.698006 (+634277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.305997 (+607991us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b5400816354. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:41.306155 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1456937728}
I20260709 13:04:41.307458  9331 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43640 (request call id 1610) took 1091 ms. Trace:
I20260709 13:04:41.307838  9331 rpcz_store.cc:276] 0709 13:04:40.216310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.459109 (+242799us) service_pool.cc:224] Handling call
0709 13:04:41.307252 (+848143us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400551c54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:41.307397 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1903036288}
I20260709 13:04:41.236527 16093 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.305902  8730 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 14.
I20260709 13:04:41.309456  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:41.310273 16093 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:41.283519 16054 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.874s	user 0.012s	sys 0.001s
I20260709 13:04:41.159436 15985 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.262429 16178 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:41.273885 16056 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:41.276242  8655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3189) took 1040 ms. Trace:
I20260709 13:04:41.319271  8655 rpcz_store.cc:276] 0709 13:04:40.235765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.247677 (+ 11912us) service_pool.cc:224] Handling call
0709 13:04:41.085565 (+837888us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.086158 (+   593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.276054 (+189896us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b54001310d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:41.276179 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1925546368}
I20260709 13:04:41.154917 16100 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:41.282253  9294 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.219049 16134 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:41.312876  8656 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:41.197989 15795 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.295473 16175 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 20
W20260709 13:04:41.331504  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:41.342006  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:41.419440  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 88009.7us
I20260709 13:04:41.419657  9324 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:41.321681  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:41.395656  9323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43664 (request call id 1378) took 1211 ms. Trace:
I20260709 13:04:41.420903  9323 rpcz_store.cc:276] 0709 13:04:40.184449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.186317 (+  1868us) service_pool.cc:224] Handling call
0709 13:04:40.186647 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.103885 (+917238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.395157 (+291272us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b5400542754. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:41.395599 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2107098368}
I20260709 13:04:41.424180  8726 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:41.307461 16182 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:41.296072 16097 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:41.344131  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62046.9us
I20260709 13:04:41.389575  8724 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:41.271103  8659 raft_consensus.cc:2468] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 22.
I20260709 13:04:41.416064  9330 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:41.438504 15814 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.442495 16188 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.443871 16176 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.444934 15890 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:41.290670 16197 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.447273 16100 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:41.450898 16126 consensus_meta.cc:287] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.572s	user 0.010s	sys 0.000s
I20260709 13:04:41.299652 16025 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.396085 16142 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:41.337313 16056 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:41.457309  8650 raft_consensus.cc:2468] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:41.458230  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.475569 16178 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.471344  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.415771  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:41.475749  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 319392us
I20260709 13:04:41.516304  9326 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:41.225277 16145 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.340394 16189 consensus_queue.cc:1048] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.134s
I20260709 13:04:41.518301  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:41.493956  8727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.012s	sys 0.010s
I20260709 13:04:41.509999  9323 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
W20260709 13:04:41.423484  8622 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:41.515322  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 348904us
I20260709 13:04:41.518555  9325 raft_consensus.cc:2393] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 22: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:04:41.522397  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 98928.3us
W20260709 13:04:41.522569  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:41.523404  8727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.012s	sys 0.010s
I20260709 13:04:41.523151  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:41.530484  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:41.537289  9324 raft_consensus.cc:2468] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 23.
W20260709 13:04:41.534976  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:41.550076  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:41.505813 16126 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:41.561618  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:41.500170 16198 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.054s
W20260709 13:04:41.567976  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:41.568890 15980 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:41.581899 16129 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:41.582458 16129 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:41.582878 16129 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:41.584206 16206 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.449s	user 0.004s	sys 0.002s
I20260709 13:04:41.584782 15538 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.585343 15538 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.588220 15538 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:41.592767 16129 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.597658 16056 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:41.601572  8696 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:41.609043  8696 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.518136 16142 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:41.573124  8725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.001s	sys 0.014s
I20260709 13:04:41.543457 16097 consensus_queue.cc:1048] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
W20260709 13:04:41.620798  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 59463.7us
I20260709 13:04:41.521309 16178 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:41.616434  8725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.002s	sys 0.014s
W20260709 13:04:41.612082  8696 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.631974  8623 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:41.632921  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.640010 16129 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.640164 15792 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:41.645147  8696 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.619475 16134 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:41.643110 16225 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:04:41.627151 16111 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.642902 16223 raft_consensus.cc:2804] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:04:41.670425  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 382733us
I20260709 13:04:41.675475 16197 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:41.645849 16228 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:41.686784  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.688056  9327 raft_consensus.cc:2410] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 13:04:41.670028 16056 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:41.633496 16126 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:41.657667  8696 leader_election.cc:336] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.659469  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:41.637735 16224 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:41.671095 16182 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.719583 16182 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.720935  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.721884  9322 raft_consensus.cc:2393] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 23: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:41.721956 16182 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:41.680006 15985 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.722824  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.724033 16178 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:41.720155 15980 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:41.726238 16175 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20.
I20260709 13:04:41.725471  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.671936 15995 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:41.679013 16223 raft_consensus.cc:697] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:41.652159  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:41.669516 16134 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.691937  9323 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:41.693343  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.691313  8696 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.709913 16224 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:41.718057  8652 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:41.668722  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 265338us
I20260709 13:04:41.731588  9324 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:41.739634  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.740585  8652 raft_consensus.cc:2368] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19.
I20260709 13:04:41.732934  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.742038  8653 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:41.733544  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.743144  8659 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:41.733845  8469 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:41.734536  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.745785  8655 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:41.690848 16025 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:41.751058  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.751689  8656 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:41.748080  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.753685  8548 leader_election.cc:400] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19.
I20260709 13:04:41.754076  8548 leader_election.cc:403] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:41.757833  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.758664  8652 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:41.764170  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 106530us
W20260709 13:04:41.765177  8696 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.765727  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:41.775849  8696 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.730687 15792 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.770054  8659 raft_consensus.cc:2468] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:04:41.782397  9324 raft_consensus.cc:2468] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 24.
W20260709 13:04:41.783357  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 233334us
I20260709 13:04:41.791473  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:41.785542  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:41.792361 15792 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:41.792770 15792 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:41.791698  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.793322 15792 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:41.792701  9322 raft_consensus.cc:2393] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 23: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:41.746941 16224 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:41.746325 15980 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:41.799566  8697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.004s	sys 0.022s
I20260709 13:04:41.774494 15985 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:04:41.736358 15995 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.720978 15991 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:41.801666  8697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.004s	sys 0.022s
W20260709 13:04:41.807983  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 622659us
I20260709 13:04:41.783170 16240 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:41.803663  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.804662  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.785063 15814 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:41.808926  8698 leader_election.cc:400] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19.
I20260709 13:04:41.809633  8697 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:41.802330  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:41.730746 16126 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.811694  9296 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:41.824230  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.821556  8698 leader_election.cc:403] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:41.821728  9327 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:41.824224  8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3200) took 1547 ms. Trace:
I20260709 13:04:41.820317  9323 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
I20260709 13:04:41.826520  8652 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:41.801615 15666 consensus_queue.cc:1059] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 13:04:41.825498  8657 rpcz_store.cc:276] 0709 13:04:40.276563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.696752 (+420189us) service_pool.cc:224] Handling call
0709 13:04:40.697033 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:40.697237 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:40.733744 (+ 36507us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.026645 (+292901us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b18001e3460. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406af72a6 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9
0709 13:04:41.454155 (+427510us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b18001e3460. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f140f9f1de1 00007f140f9fe5b1 00007f1406b0be07 00007f140f966ab9 00007f1406b2869b 00007f1406b2693f 00007f1406b26455 00007f1409c0e060 00007f1409c20716 00007f1409c31287 00007f1409c31208
0709 13:04:41.823764 (+369609us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.823818 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:41.823866 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.824000 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.824163 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":35823,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2809,"raft.queue_time_us":76576,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":55499264,"thread_start_us":358285,"threads_started":1}
I20260709 13:04:41.831375  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:41.801862 16198 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.019s
I20260709 13:04:41.803987 15980 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.823125 16142 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.823822 16100 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:41.824985 16223 consensus_queue.cc:237] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.823453 16111 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:41.835029 15855 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:41.843355 16189 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:41.832538 16240 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:41.829651 16203 raft_consensus.cc:3060] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:41.877861  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:41.805553 16227 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 13:04:41.838773 15538 raft_consensus.cc:2804] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:41.852538 15814 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 13:04:41.879551 16197 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:41.867045 16126 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.882547 16175 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.883955  8697 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:41.855010 15991 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.886478  8656 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:41.888192  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.863214 15995 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:41.848177 16182 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:41.890424  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:41.899169  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 116610us
I20260709 13:04:41.877957 16240 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:41.888873  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.859390 15980 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:41.887951 16215 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:41.935691  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.935873  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:41.937223  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 53388.9us
I20260709 13:04:41.902264  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.949724  8652 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 19: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:41.942284  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.950736  8658 raft_consensus.cc:2393] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 24: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:41.859818 16111 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.950876  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:41.951627  8659 raft_consensus.cc:2368] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21.
I20260709 13:04:41.952633  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.902171  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.931713 15538 raft_consensus.cc:697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:41.950505 16215 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:41.970022 16215 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:41.970199 15538 consensus_queue.cc:237] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.973798  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:41.974543  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 451393us
W20260709 13:04:41.975243  8622 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:41.975883  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:41.976709  8658 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
W20260709 13:04:41.980556 16217 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.413s	user 0.000s	sys 0.002s
I20260709 13:04:41.991052  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:41.991838 15814 consensus_queue.cc:1059] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 13:04:41.992383 16244 log.cc:927] Time spent T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 13:04:41.992756 16178 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:41.993022 15814 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:41.993542  8653 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:41.994086 15814 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:41.994474 15814 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:41.996212 15814 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:41.997490 15814 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:41.933869 16197 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:42.003909 16242 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:04:41.936529 16182 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:42.006765 16182 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:41.936987 16175 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:41.947556 16224 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.017835 16224 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:41.939374 16041 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:41.944532 16240 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:42.024217  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.025283  9328 raft_consensus.cc:2368] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20.
W20260709 13:04:42.066673  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:42.067173  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 288795us
W20260709 13:04:42.067806  8696 leader_election.cc:336] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.071700  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.073770  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.076248 15795 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:42.087802  8726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40408 (request call id 3290) took 1154 ms. Trace:
I20260709 13:04:42.088347  8726 rpcz_store.cc:276] 0709 13:04:40.933159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.933475 (+   316us) service_pool.cc:224] Handling call
0709 13:04:40.941138 (+  7663us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.455944 (+514806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:42.087464 (+631520us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b5400338e54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:42.087709 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1093102336}
I20260709 13:04:42.094187 16188 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:42.094648 16188 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:42.095009  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:42.095494 16188 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.096789 16188 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:42.099469 16189 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19.
W20260709 13:04:42.101576  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 110602us
I20260709 13:04:42.102761 16127 consensus_queue.cc:260] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.104297 16127 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:42.105214 16145 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:42.112668  8550 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.113793  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.095938 16186 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:42.115010 16186 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:42.115415 16186 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:42.109979  8698 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:42.118522  8698 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:42.119663  8698 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:42.101147 16111 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:42.125790  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.121111  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:42.127238  9325 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:42.131991  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.132123  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:42.132827  8623 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:42.127756  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.139837  8653 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:42.132889  9328 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:42.123739  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55970.6us
I20260709 13:04:42.123564  8655 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 22.
I20260709 13:04:42.140678  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.142215  8657 raft_consensus.cc:1240] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:42.135761 16249 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:42.143571  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:42.144531  8476 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 207) took 31637 ms (client timeout 15000 ms). Trace:
I20260709 13:04:42.124571 16041 raft_consensus.cc:2804] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:42.145655 16156 raft_consensus.cc:3060] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:42.145967 16041 raft_consensus.cc:697] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:42.147303  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:42.147480  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.147984  9329 raft_consensus.cc:3060] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:42.134572 16175 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:42.121444 15795 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.126313 16100 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:42.122987 15890 raft_consensus.cc:2804] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:42.122743 16145 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:42.150472  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:42.189180 15995 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:42.190482 15995 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:42.190886 15995 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:42.192510  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:42.194710  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.177955  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.199939  8725 raft_consensus.cc:2393] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 22: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:42.148756  8550 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:42.201164  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 371735us
I20260709 13:04:42.202167  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.205732  9324 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 19: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:42.183343  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:42.211262  9329 raft_consensus.cc:2468] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:42.211580  8731 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:42.182225  8698 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:42.121479  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 146262us
W20260709 13:04:42.242594  8622 leader_election.cc:336] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:42.234905  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 52753.6us
W20260709 13:04:42.245760  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.243432  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:42.247285  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 98589.4us
I20260709 13:04:42.248551  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.245608  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:42.250908  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 48847.2us
I20260709 13:04:42.194576 16201 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.252805 16201 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.255669 16201 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:42.257253 16197 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:42.144933  8476 rpcz_store.cc:269] 0709 13:04:10.510162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.386027 (+2875865us) service_pool.cc:224] Handling call
0709 13:04:42.144462 (+28758435us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:37.641883 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:37.647120 (+  5237us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:37.647177 (+    57us) write_op.cc:435] Acquired schema lock
0709 13:04:37.647189 (+    12us) tablet.cc:662] Decoding operations
0709 13:04:37.694618 (+ 47429us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:37.694719 (+   101us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:37.694756 (+    37us) tablet.cc:685] Acquiring locks for 23 operations
0709 13:04:37.696690 (+  1934us) tablet.cc:701] Row locks acquired
0709 13:04:37.696738 (+    48us) write_op.cc:260] PREPARE: finished
0709 13:04:37.696943 (+   205us) write_op.cc:270] Start()
0709 13:04:37.697122 (+   179us) write_op.cc:276] Timestamp: P: 1783602277696870 usec, L: 0
0709 13:04:37.697162 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 13:04:37.711051 (+ 13889us) log.cc:844] Serialized 8817 byte log entry
0709 13:04:38.293948 (+582897us) op_driver.cc:464] REPLICATION: finished
0709 13:04:38.533595 (+239647us) write_op.cc:301] APPLY: starting
0709 13:04:38.543877 (+ 10282us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:38.630673 (+ 86796us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:38.630794 (+   121us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:39.007890 (+377096us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:39.008332 (+   442us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=46,key_file_lookups=46,delta_file_lookups=46,mrs_lookups=0
0709 13:04:39.008394 (+    62us) write_op.cc:312] APPLY: finished
0709 13:04:39.008908 (+   514us) log.cc:844] Serialized 204 byte log entry
0709 13:04:39.009112 (+   204us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:39.009327 (+   215us) write_op.cc:454] Released schema lock
0709 13:04:39.009451 (+   124us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:39.009907 (+   456us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:39.009939 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":221811,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":234041,"apply.run_cpu_time_us":38326,"apply.run_wall_time_us":476314,"cfile_cache_hit":94,"cfile_cache_hit_bytes":100221,"num_ops":23,"prepare.queue_time_us":552385,"prepare.run_cpu_time_us":15790,"prepare.run_wall_time_us":92919,"raft.queue_time_us":510396,"raft.run_cpu_time_us":8043,"raft.run_wall_time_us":236166,"replication_time_us":572476,"spinlock_wait_cycles":33664,"thread_start_us":812065,"threads_started":4,"wal-append.queue_time_us":15629,"wal-append.run_cpu_time_us":18040,"wal-append.run_wall_time_us":2326545}]]}
W20260709 13:04:42.248553  8622 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:42.272313  8622 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:42.273727  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:42.274662  8622 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:42.275766  8622 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.276297  8622 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:42.280294  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.177148 16129 raft_consensus.cc:2804] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:42.281014  8725 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:42.193562 16100 raft_consensus.cc:697] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:42.178463 16249 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:42.287642  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.204624 15890 raft_consensus.cc:697] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:42.191748 15995 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.242326 16041 consensus_queue.cc:237] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.182521 15795 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.322150  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.463258  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:42.475673  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 188118us
I20260709 13:04:42.476047  8729 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:42.248135 16145 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 13:04:42.394707 15666 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.031s	sys 0.044s
I20260709 13:04:42.413782  8731 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 15.
W20260709 13:04:42.477736 15666 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.031s	sys 0.062s
I20260709 13:04:42.478416 15666 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.332377 16156 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:42.478957 15666 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.443634 16175 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:42.324288 16215 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.479768 16156 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:42.480405 16175 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.279569 16223 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:42.317747 16129 raft_consensus.cc:697] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 24 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:42.482761 16175 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:42.318333 15792 consensus_meta.cc:287] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.524s	user 0.009s	sys 0.010s
I20260709 13:04:42.424268 16134 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:42.288466 16213 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:42.423030 16100 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.478585  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:42.553078  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.553851  8731 raft_consensus.cc:2393] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:42.477039  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.554395  8729 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:42.326807  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:42.482327 16215 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:42.476981  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:42.318215  8476 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 307) took 17674 ms (client timeout 15000 ms). Trace:
W20260709 13:04:42.556213  8476 rpcz_store.cc:269] 0709 13:04:24.643806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:42.317945 (+17674139us) service_pool.cc:211] Skipping call since client already timed out
0709 13:04:42.318132 (+   187us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:04:42.512445 16129 consensus_queue.cc:237] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.513500  8726 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 23.
I20260709 13:04:42.294193  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.558642  8725 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:42.559418  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:42.560729  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.560948  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.561134  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.562000  8733 raft_consensus.cc:2410] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 13:04:42.561865  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.562175  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.562876  8725 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:04:42.563946  8733 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:42.451676  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:42.381072  8548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.014s	sys 0.005s
I20260709 13:04:42.261476 16054 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:42.481257 15666 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:42.566250  8548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.015s	sys 0.006s
W20260709 13:04:42.566696  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 734409us
I20260709 13:04:42.566951  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:42.567368  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:42.568224 15666 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:42.568652 15666 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:42.569070 15666 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:42.551482 15792 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.578538  8658 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:42.580017 15792 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:42.296878  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:42.410506 16189 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.291257 16201 raft_consensus.cc:3060] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:42.534096  8654 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:42.476867  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:42.343487 16095 raft_consensus.cc:3055] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:04:42.530045 16203 consensus_meta.cc:287] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.646s	user 0.008s	sys 0.006s
I20260709 13:04:42.287742 15991 consensus_queue.cc:1059] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.255s
W20260709 13:04:42.541530  9330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.015s	sys 0.004s
W20260709 13:04:42.594393  9330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.015s	sys 0.004s
W20260709 13:04:42.549659  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 267018us
I20260709 13:04:42.595677  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.595878 15991 raft_consensus.cc:3055] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:42.596508 15991 raft_consensus.cc:740] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:42.591653  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 449743us
I20260709 13:04:42.590838 16213 raft_consensus.cc:697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:42.591212 16095 raft_consensus.cc:740] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:42.332427 15985 consensus_meta.cc:287] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.002s
I20260709 13:04:42.597142  8549 leader_election.cc:400] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21.
I20260709 13:04:42.609982  8549 leader_election.cc:403] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
W20260709 13:04:42.612067  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:42.599458 16250 raft_consensus.cc:3060] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:42.598424  9322 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:42.598846  9331 raft_consensus.cc:1240] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEADER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:42.595201  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 118458us
I20260709 13:04:42.602098  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:42.617259  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.606448  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:42.606827  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:42.607347  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:42.607525  9329 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:42.593364 16203 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:42.326419 16240 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.613806  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:42.595794 16253 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:04:42.616031  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.623526  9294 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.591953 16189 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.632649  8697 leader_election.cc:400] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 19. Current term is 20.
I20260709 13:04:42.636534  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:42.619323 16182 consensus_meta.cc:287] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.612s	user 0.004s	sys 0.010s
I20260709 13:04:42.585152 15890 consensus_queue.cc:237] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.613231 15985 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19.
I20260709 13:04:42.617379 16245 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.632236 16253 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:42.642900 16127 raft_consensus.cc:2804] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:42.632794 16264 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:42.643625 16263 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:42.651803 16203 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:42.689257  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:42.689586  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 286564us
W20260709 13:04:42.665592 16186 consensus_meta.cc:287] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.550s	user 0.012s	sys 0.000s
W20260709 13:04:42.674290 16188 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.546s	user 0.007s	sys 0.003s
I20260709 13:04:42.690541  8625 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:42.646229 16054 consensus_queue.cc:1059] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 13:04:42.655504 16095 consensus_queue.cc:260] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.706728 16229 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:04:42.707597 16258 log.cc:927] Time spent T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20260709 13:04:42.710692 16221 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 1.159s	user 0.006s	sys 0.001s
W20260709 13:04:42.746182  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 470422us
W20260709 13:04:42.747576  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 131562us
I20260709 13:04:42.652218 16182 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.679543 16127 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:42.748584  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:42.749650  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 131357us
I20260709 13:04:42.673979 16253 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:42.702710  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:42.680696 16267 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:42.752200  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 61788.4us
I20260709 13:04:42.789903  8550 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.790442  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:42.791591  8697 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:42.694692 15980 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:42.751502  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 47854.4us
I20260709 13:04:42.788265  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:42.674228 16245 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.624631 15991 consensus_queue.cc:260] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.753217 16156 raft_consensus.cc:3055] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:04:42.792002 16251 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260709 13:04:42.767644  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.792348  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:42.820562  9327 raft_consensus.cc:3060] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:42.702840 15705 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:42.789656 16127 raft_consensus.cc:3060] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:42.821347 15705 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.824421 15705 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:42.825011 15705 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:42.612663 16249 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:42.826187 16249 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.786509 16224 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:42.786693 16215 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:42.831238  8699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.836421  9327 raft_consensus.cc:2468] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:42.705559 16186 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.811684 15792 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:42.838078 16228 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:42.838392 16215 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:42.839752 16265 log.cc:927] Time spent T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
I20260709 13:04:42.815410 15980 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.840862 15795 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:42.843218 16197 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:42.702657 16203 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.818104 16156 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:42.819074 15985 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:42.865806  9328 raft_consensus.cc:2368] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 22. Current term is 23.
I20260709 13:04:42.831466 15705 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:42.850677 15991 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:42.686818 16213 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.854300 16134 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:42.878214 15705 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:42.913748 16254 log.cc:927] Time spent T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 13:04:42.915933  9330 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1611) took 1235 ms. Trace:
I20260709 13:04:42.916592  9330 rpcz_store.cc:276] 0709 13:04:41.683800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.684547 (+   747us) service_pool.cc:224] Handling call
0709 13:04:41.685051 (+   504us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:41.830340 (+145289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.830392 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:41.876606 (+ 46214us) log.cc:844] Serialized 26 byte log entry
0709 13:04:42.594937 (+718331us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:42.594977 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:42.595055 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:42.915545 (+320490us) raft_consensus.cc:1685] finished
0709 13:04:42.915586 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:42.915810 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33373,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":59,"thread_start_us":800175,"threads_started":2,"wal-append.queue_time_us":747684}
W20260709 13:04:42.923746  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 128284us
W20260709 13:04:42.916492 16269 log.cc:927] Time spent T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
I20260709 13:04:42.862421 15666 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:42.932901  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 139468us
W20260709 13:04:42.936021  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:42.919777 16065 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.923448 16054 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:42.917610 16134 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:42.933251 16156 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:42.968798 16261 log.cc:927] Time spent T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 13:04:42.976097 15980 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:42.986428  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 170319us
I20260709 13:04:42.995790 16156 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:42.996520 16273 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:04:43.000258 16224 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:42.961721 16054 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.002822 16240 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.003259 16054 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.004184 15666 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.006031 16134 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:43.007864 16127 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:42.993091 15980 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:43.008986 16253 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.014969 16245 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.018776 16253 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.021764 16182 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:43.023751 16056 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:43.024544 16056 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.026214 16186 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:42.953675 16065 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.029987 16126 raft_consensus.cc:3055] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:43.030548 16126 raft_consensus.cc:740] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:43.031603 16126 consensus_queue.cc:260] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:43.034075 15814 consensus_meta.cc:287] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 1.032s	user 0.016s	sys 0.000s
I20260709 13:04:43.039415 16095 raft_consensus.cc:3060] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:43.011606 15980 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.043929 15980 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.012434 16215 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:43.049814  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:43.057322  9294 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:43.058393  9294 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:43.059448  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 267925us
I20260709 13:04:43.029127 16065 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.061676  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:43.047437 16268 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
W20260709 13:04:43.063577  8696 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:43.061940  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 124273us
W20260709 13:04:43.066725  8469 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 15 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:43.067763  8623 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:43.080946  8728 raft_consensus.cc:1275] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 13:04:43.082741  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:43.088090  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 265315us
I20260709 13:04:43.088294  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.092139  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.102190  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.102895  8657 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:43.122135  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.122913  8725 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:43.125576 16186 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:43.126186 16186 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.128499 16186 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:43.129000 16186 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:43.130645  8475 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 13 to 14, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 14 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:43.138844  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.087275 16224 raft_consensus.cc:2804] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:43.139660  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:43.140714  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48849.4us
I20260709 13:04:43.141597  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.154210  8727 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:04:43.155181  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.155915  8727 raft_consensus.cc:2393] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:43.143410  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.139648 16203 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.157859 16280 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:43.160065  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.160275 16240 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.160897 16240 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.148501  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:43.151659  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.152031  8731 raft_consensus.cc:1240] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:43.143069  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:43.164774  8730 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:43.142133  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.166285  8726 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 14.
I20260709 13:04:43.141268  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.142292  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.145103  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:43.171197  8623 leader_election.cc:400] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 22. Current term is 23.
I20260709 13:04:43.171623  8623 leader_election.cc:403] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:43.149840  8729 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
W20260709 13:04:43.172401  9294 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.152905  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:43.183362  9331 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:43.162951  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.179258  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:43.140884  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 52044.7us
I20260709 13:04:43.188850  8625 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:43.193990  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51778.7us
I20260709 13:04:43.194527  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:43.195437  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.196693  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.197328  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.176198 15792 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:04:43.208498  8550 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.211273  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.211194  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.172519 16127 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.213770  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.214560  8727 raft_consensus.cc:2393] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 15: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:43.174494 16224 raft_consensus.cc:697] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:43.224876 16224 consensus_queue.cc:237] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.216686 16282 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:43.213624 16025 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:43.217032 16253 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:43.151985 16186 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.208280 16178 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:43.270668  8654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38104 (request call id 3630) took 1146 ms. Trace:
I20260709 13:04:43.210740 16111 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:04:43.271045  8549 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:43.218900 16264 consensus_meta.cc:287] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.523s	user 0.009s	sys 0.000s
I20260709 13:04:43.271341  8654 rpcz_store.cc:276] 0709 13:04:42.124362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:42.125657 (+  1295us) service_pool.cc:224] Handling call
0709 13:04:42.532820 (+407163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:42.593650 (+ 60830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:43.270401 (+676751us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54003223d0. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c158ad 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:43.270602 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":935686528}
I20260709 13:04:43.207736 16285 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:43.213791 16280 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:43.216516  8550 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:43.223863  9294 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.226866  8625 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.186998 16245 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:43.192776 16275 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:43.313609 16275 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.314049 16275 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:43.197357 16054 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.271250 16178 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.314406  8728 raft_consensus.cc:1240] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:43.217335 16263 consensus_queue.cc:1048] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:43.315148 16178 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:43.264511  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51454.6us
I20260709 13:04:43.259580  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.319736  9297 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:43.321640  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 105170us
W20260709 13:04:43.321810  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 95033.1us
I20260709 13:04:43.321707  9297 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:43.318658 16263 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:43.325541 16263 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.307487 16264 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929) with higher term. Message: Invalid argument: T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21.
I20260709 13:04:43.308490 16285 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:43.328047 16189 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.328380 16285 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:43.270658 16145 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:43.223821 16250 consensus_meta.cc:287] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.600s	user 0.005s	sys 0.005s
I20260709 13:04:43.133131  8475 catalog_manager.cc:5697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 6 to 12, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:43.189131 15705 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:43.310209  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 86387.5us
I20260709 13:04:43.207839 16134 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:43.330288  8550 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.332911 16266 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.312682 16111 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:43.346011 16111 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.346972 16186 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:43.330415 16201 consensus_meta.cc:287] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.736s	user 0.010s	sys 0.002s
I20260709 13:04:43.284876  9324 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 16.
I20260709 13:04:43.331992  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.330119 16065 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:43.331761  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
W20260709 13:04:43.336428  9294 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:43.359110  9294 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.327952 16263 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.358842  9297 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:43.330317 15980 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:43.362831  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:43.333047 16245 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:43.335034 16182 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:43.363698 16186 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:43.373678 16186 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.374112 16186 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:43.374676 16266 consensus_queue.cc:1059] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:43.376205 16127 raft_consensus.cc:3055] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:43.376633 16127 raft_consensus.cc:740] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:43.367878  9294 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.377629 16127 consensus_queue.cc:260] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.337764 16250 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:43.372752 15538 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:43.379199 16127 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:43.379606 15538 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.380043 15538 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:43.381799 16289 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.162s	user 0.002s	sys 0.000s
I20260709 13:04:43.383656  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7
W20260709 13:04:43.384243  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 312367us
I20260709 13:04:43.386878 16203 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:43.387562 16203 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:43.387979 16126 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:43.393622 16056 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.366353 16245 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:43.394780 15980 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:43.395108 16245 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:43.395864 15980 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.398164 15705 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.370430 16134 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:43.399148 16134 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.399632 16134 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:43.399781  8730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42702 (request call id 1419) took 1092 ms. Trace:
I20260709 13:04:43.400256  8730 rpcz_store.cc:276] 0709 13:04:42.307752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:42.308656 (+   904us) service_pool.cc:224] Handling call
0709 13:04:43.399584 (+1090928us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b5400436f54. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c1a2fa 00007f140ebdbb17 00007f14081566d4 00007f1408156662 00007f1408156591 00007f140815650c 00007f14081557c2 00007f1407f5148c 00007f1407f508c6 00007f1407f5026f
0709 13:04:43.399712 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1103151872}
I20260709 13:04:43.394534 15995 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:43.337594 16145 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.406313 16145 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.336966 16280 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 13:04:43.406378 16240 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.408527 16182 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.377530 16249 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:43.410756 16186 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.415067 16186 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.399088 15538 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.410669 16134 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.417577  9327 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:43.369771  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.419384 16134 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.417640 15538 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.410991 16182 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:43.428263  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 58596.4us
I20260709 13:04:43.429147  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.429544 16267 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:43.433491  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.434201  9330 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 16.
I20260709 13:04:43.434669  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.445156  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:43.450491  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.451191  9330 raft_consensus.cc:2468] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:43.388602 16203 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.365939 16201 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:43.488085  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:43.452718 16126 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:43.487422 16145 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.489320 16056 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:04:43.512132  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 178820us
I20260709 13:04:43.388677  8475 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 9 to 12, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:43.521359  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.499361 16156 raft_consensus.cc:2804] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:43.526995  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.491815 16280 consensus_queue.cc:1059] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 13:04:43.531093  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.534487  8725 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:43.517381 16056 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.499830 16178 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.526645 16156 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:43.536118 16292 log.cc:927] Time spent T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.230s	user 0.002s	sys 0.000s
W20260709 13:04:43.548631  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 184051us
I20260709 13:04:43.549840  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.552984  8732 raft_consensus.cc:2393] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 21: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:43.528401 16227 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:43.556874  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.543533 16056 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:43.519160 16145 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.568177  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.569227  8731 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:43.531355 15538 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 13:04:43.570818 15538 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:43.571193 15538 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:43.572013 15538 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.574615 15985 raft_consensus.cc:3055] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:43.575028 15985 raft_consensus.cc:740] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:43.575853 15985 consensus_queue.cc:260] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.555218 16156 raft_consensus.cc:3060] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:43.591341  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.588560  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.592056  8727 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:43.592645  8731 raft_consensus.cc:3060] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:43.593233  9297 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:43.598242  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:43.598470  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:43.599622  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.599933  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.600651  8657 raft_consensus.cc:1275] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 13:04:43.596741  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.600967  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.563948 15792 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 22. Current term is 23.
I20260709 13:04:43.602444  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:43.604148  8654 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:43.602799  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:43.605763  8728 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:04:43.607620  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:43.608567  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.608803  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:43.610117  8655 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:43.611276  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.585228 16249 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.611989  8652 raft_consensus.cc:2393] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:43.611613  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.618604  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.608204  8657 raft_consensus.cc:2376] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19. Re-sending same reply.
I20260709 13:04:43.625530 16020 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.626097 16020 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.628515 16020 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.629693  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:43.632949  8622 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.635236  8622 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:43.636270  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.643373  9296 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.643790  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.644577  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.645082  8657 raft_consensus.cc:1240] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:43.636870 15705 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.645570 16300 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:43.645819 15705 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:43.646270 16300 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:43.646521 16275 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:43.646942 16300 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:43.647534  8729 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:43.647953 15705 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.648479 15705 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:43.637914  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:43.650560  8651 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:43.638062 16129 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.638341 16298 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:43.637092  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:43.654260  8698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:43.562026 16227 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:43.641122  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:43.651571 16129 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:43.654625  8625 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:43.640269  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:43.651664  8650 raft_consensus.cc:2468] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 23.
W20260709 13:04:43.655560  8622 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 127 similar messages]
I20260709 13:04:43.656271  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:43.657115  8728 raft_consensus.cc:2468] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
W20260709 13:04:43.659480  9294 leader_election.cc:336] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.662577  9294 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:43.663718  9294 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.664135  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.667784  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.668433  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.668681  8729 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:43.665175  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:43.652400 16298 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:43.745674  9297 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:43.637259 15795 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:43.747756  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 52158.1us
I20260709 13:04:43.748893  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:43.758761  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:43.759824  9323 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:43.763245  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:43.778827  9324 raft_consensus.cc:1275] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:04:43.666316 16020 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:43.675344 16182 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:43.675084 16126 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:43.794658  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 126291us
I20260709 13:04:43.746482 16134 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:43.800568  8731 raft_consensus.cc:2468] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:04:43.802980  8654 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
I20260709 13:04:43.659319 15855 consensus_queue.cc:1059] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.095s
I20260709 13:04:43.700210 15792 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:43.754410 16304 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 13:04:43.822688  8549 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:43.679020 16300 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.746166 16253 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.661334 16295 consensus_queue.cc:1048] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.132s
I20260709 13:04:43.791253 16237 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:43.779948 16186 raft_consensus.cc:2804] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:43.821918 16203 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:43.787915 15795 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:43.660297 16249 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.831291 15792 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:43.845655 16237 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.849674 15666 raft_consensus.cc:3055] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:43.833220 16280 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 13:04:43.859879 15792 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:43.898545 16303 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.276s	user 0.000s	sys 0.000s
I20260709 13:04:43.853996 16186 raft_consensus.cc:697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:43.852741 16300 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:43.854487 16203 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.943326 16298 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.860631 16178 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.900934 16237 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:43.864476 16129 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.950970 16203 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:43.930150 16245 consensus_meta.cc:287] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.534s	user 0.010s	sys 0.000s
W20260709 13:04:43.956848 16301 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.135s	user 0.001s	sys 0.001s
W20260709 13:04:43.967451 16285 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.639s	user 0.011s	sys 0.001s
I20260709 13:04:43.963517 16245 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.967880 16285 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.968050 16249 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.969938 16285 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:43.970158 16245 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.970664 16186 consensus_queue.cc:237] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.965669 16275 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:43.955808 16300 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:43.976158 16300 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:43.612015  8475 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 8 to 11, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 11 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:44.010545  8548 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:44.013314  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 190571us
I20260709 13:04:44.019269  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:43.835639  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 193326us
W20260709 13:04:44.023805  8622 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.025216  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 257261us
I20260709 13:04:44.026602  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:44.017158  8475 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 2 to 10, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 10 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:43.962842 16145 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.036176 16134 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:44.036355 15855 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:43.902009 15666 raft_consensus.cc:740] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:44.037539 16129 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.044669 15795 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:43.973232 16227 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:44.033202  8622 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.093327  8622 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:44.095661  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 62594.9us
I20260709 13:04:44.086480  9297 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:44.096777 16224 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:44.048405 16127 raft_consensus.cc:3060] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:44.101792  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 72176.5us
I20260709 13:04:44.126966  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:44.128228 16266 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:44.096848  8622 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.131500  8622 leader_election.cc:336] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.132301  8622 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:44.109378 16300 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.114238 16129 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.115815 16249 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:44.122941 15666 consensus_queue.cc:260] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.102115 15995 raft_consensus.cc:2804] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:44.142074 16213 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
W20260709 13:04:44.102203 16025 consensus_meta.cc:287] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.842s	user 0.008s	sys 0.003s
W20260709 13:04:44.154446 16268 log.cc:927] Time spent T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 13:04:44.158454 15538 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:44.165648 16213 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.169217 16025 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:44.170991 16319 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
W20260709 13:04:44.173719  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 355406us
I20260709 13:04:44.180279 16134 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.183144 16300 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.183209 16224 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:44.180843 16129 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.183634 15995 raft_consensus.cc:697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:44.188236 15995 consensus_queue.cc:237] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:44.183830  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 52548.6us
I20260709 13:04:44.198522 16186 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.168603 16213 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:44.201212 16186 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.204666  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:44.205651 16201 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.206457 16186 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:44.207450 16224 consensus_queue.cc:1048] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:44.211970  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:44.216346  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:44.217331 16300 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:44.219398 16182 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:44.227625 16249 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:44.229228  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.195653 16266 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 13:04:44.245010 16224 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:44.245869 16224 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.249682 16224 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:44.217617 16025 consensus_queue.cc:1059] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
W20260709 13:04:44.250334  8727 consensus_meta.cc:287] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.643s	user 0.011s	sys 0.006s
I20260709 13:04:44.251118 16025 raft_consensus.cc:3055] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:44.251236  8727 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
I20260709 13:04:44.251914 16025 raft_consensus.cc:740] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:44.255865 15985 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:44.253819 16025 consensus_queue.cc:260] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.258298 15666 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:44.260648 16025 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:44.261302  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.206508 16201 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.271078 16201 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:44.273525  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 69081.1us
W20260709 13:04:44.275213  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 762742us
W20260709 13:04:44.311358  8696 leader_election.cc:336] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.312093  8696 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:44.303555  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:44.317269  9296 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:44.318735 16300 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:44.319329 16300 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:44.320326 16300 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:44.284452 16156 consensus_meta.cc:287] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.682s	user 0.008s	sys 0.017s
I20260709 13:04:44.322610 16156 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.272222 16282 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:04:44.237203  8622 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.302178  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.303634  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:44.308465  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 96644.4us
I20260709 13:04:44.321944 16321 raft_consensus.cc:2804] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:44.334048 16321 raft_consensus.cc:697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:44.335255 16321 consensus_queue.cc:237] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.332512  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.328457 16282 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:44.338762 16282 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:44.344676  8622 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:44.346030  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 125880us
I20260709 13:04:44.349247 16298 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:44.350538 16295 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.351176 16295 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.352401 16127 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:44.353650 16295 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:44.374145 16041 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:44.375684 16237 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:44.376224  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 65022us
I20260709 13:04:44.377060  8653 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:44.378120  8696 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.383906 16056 consensus_meta.cc:287] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Time spent flushing consensus metadata: real 0.801s	user 0.004s	sys 0.018s
I20260709 13:04:44.384487 16056 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:44.391822  9294 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.392653  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:44.394815 16134 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:44.396479  8650 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:44.396808 16213 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:44.397559 16213 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.400908 16213 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:44.408591  9294 leader_election.cc:336] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.381910 16237 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.443358  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:44.449648  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45329.5us
I20260709 13:04:44.454391  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.457919 16156 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.464017  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.437963 16127 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.465211  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:44.466256  8730 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:44.466809  8732 raft_consensus.cc:1240] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:04:44.466329 16127 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.464358  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:44.472522  8658 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:44.475168 16056 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:44.480844  8696 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.481702  8696 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:44.483567  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 45057.3us
W20260709 13:04:44.484982  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:44.486927  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 169816us
I20260709 13:04:44.502795  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:44.505792  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 102017us
I20260709 13:04:44.507670  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.508694  8656 raft_consensus.cc:2393] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 18: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
W20260709 13:04:44.507732  9294 leader_election.cc:336] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.511801 15792 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: Time spent flushing consensus metadata: real 0.581s	user 0.011s	sys 0.005s
I20260709 13:04:44.512451 15792 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.513885 15666 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.517592 15666 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.519932  8728 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:44.529776 16025 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.530568 16025 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.532986 16025 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:44.558856  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.567685  8730 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:44.571519  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:44.564360  8696 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.583859  8622 leader_election.cc:336] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.584390  8622 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:44.584527 15666 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:44.586869  9296 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.543716 16056 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:44.590864  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:44.591848  8727 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:44.533665 15792 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:44.599049  8731 raft_consensus.cc:2393] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 16: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:44.601616  8658 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:44.580647 16127 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.634594  9331 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:04:44.638088  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.638353  9322 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:04:44.618563 15995 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:44.640172 15995 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.620777  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.642874  8658 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:44.644680  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:44.645364  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.646068  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:44.646121  8729 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 17.
I20260709 13:04:44.642858 15995 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:44.646289  8724 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:44.647421  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.647583  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.648181  8725 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:44.649369 16266 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:44.646135  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 123798us
I20260709 13:04:44.650228 16266 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.630496  9330 raft_consensus.cc:1275] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:04:44.651945 16056 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:44.653352  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:44.655053 16227 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:44.655908  9326 raft_consensus.cc:2393] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:44.659452 16266 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.661089 16266 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.662075 16266 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.666302 16266 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.673226  8728 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:44.673799  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.613603 16237 raft_consensus.cc:2749] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:44.650614  8696 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.631153  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47384.4us
W20260709 13:04:44.683902  9294 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.694610  9294 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.698216 16321 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.703410  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.690137 15538 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.699191 16321 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.708078 16266 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.708946 16266 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.705237 15538 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:44.702023  9294 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.715965  9294 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.690654 16127 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:44.684048 15995 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:44.689560 16282 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:44.696259  8622 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.745308  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.751374  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:44.751374  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:44.758771  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.764307  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:44.744138 16127 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.808212 16025 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.808810 16025 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:44.810652  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:44.811302 16025 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.813158  8728 raft_consensus.cc:1275] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 13:04:44.815325 16266 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.816839 16266 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.817206  8698 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:44.817323 16266 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:44.818014  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 126238us
I20260709 13:04:44.819517 16095 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:44.820035 16095 raft_consensus.cc:697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:44.820909 16095 consensus_queue.cc:237] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.821758 16266 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:44.824400  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:44.826051 16285 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.826782 16285 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:44.838773  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:44.756402 16224 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 13:04:44.841504 16321 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:44.845546  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:44.848356 16329 log.cc:927] Time spent T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.098s	user 0.001s	sys 0.001s
W20260709 13:04:44.854389  9294 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.854530  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.854871  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:44.855149  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.855358  9329 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:44.856318  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.856761  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.857673  9326 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:44.858717  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:44.860526  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.836273 15538 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:44.866689  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:44.869375  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:44.871390 15792 raft_consensus.cc:493] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.756129 16298 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.116s
I20260709 13:04:44.872138 15792 raft_consensus.cc:515] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.871433  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.872164  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.875394  8655 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:44.875578  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.876480  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.876590 15792 leader_election.cc:290] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.877157  8656 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:44.872990  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.878509  8651 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:44.804791  8623 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:44.747911 15995 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:44.884438  8657 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:44.884892  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 80236.8us
I20260709 13:04:44.888373 16237 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:44.893440 16282 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:44.893904  8548 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:44.903708  9330 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
W20260709 13:04:44.915345  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 223064us
W20260709 13:04:44.916042  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 265488us
I20260709 13:04:44.915764  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.915242 16285 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:44.933641  8696 leader_election.cc:336] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.917232 16249 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:44.939421  8622 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.940161  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.951396  9323 raft_consensus.cc:1275] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:04:44.902001 16237 consensus_queue.cc:1048] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
W20260709 13:04:44.954923  8548 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 61050.4us
W20260709 13:04:44.962325  8550 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.958927  8622 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:44.976684  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:44.980499  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:44.984127  8548 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:44.982625  9331 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:44.989837  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:44.990579  9323 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:44.990667 16330 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.148s	user 0.000s	sys 0.009s
I20260709 13:04:44.991724  9326 raft_consensus.cc:1275] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:44.957621 16249 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.984970  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:44.995324  8657 raft_consensus.cc:2468] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 22.
I20260709 13:04:45.006145  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:45.009963  8696 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:45.006664  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:45.019578  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.020186 16127 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.021456  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.022190  8727 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:45.022404 16249 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.023586  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.024837 16249 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.000026 16266 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:45.025595  8696 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:45.025718 16266 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.026305 16266 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:45.027099  8696 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:45.027626  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.028743 15705 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:45.027957  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.030019 16300 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:45.030361 16283 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:45.031487  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:45.032527 16189 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:45.033779  9297 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:45.035246 16025 consensus_queue.cc:1048] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:45.036098 16300 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:45.036749 16025 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:45.037678  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.037583 16300 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:45.037768  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:45.010169 16056 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.039377 16186 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:45.039259 16056 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.039722 16285 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:45.039878 16186 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:45.041291 16186 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.011421 15985 consensus_queue.cc:1059] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 13:04:45.037189 16025 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.049154  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.044693 16186 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:45.049618 16025 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:45.059532  8655 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:45.061499 16127 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.062135 16127 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.065038 16127 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.057332 16056 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.100178  9323 raft_consensus.cc:2468] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:45.057883  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.058301  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.060987  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.066118 15995 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.067371  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.067834  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.037583 15705 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:45.070923  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:45.066111  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2142382a5d4439915e07c70837c688" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.115064  9324 raft_consensus.cc:1240] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 13:04:45.056200  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.105702  8655 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:45.108055  9325 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
W20260709 13:04:45.072714  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:45.116175  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 43506.5us
W20260709 13:04:45.072721  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 145043us
I20260709 13:04:45.120891  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.124322  8656 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:45.134454  8698 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:45.134786 16290 raft_consensus.cc:3055] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:45.135309 16290 raft_consensus.cc:740] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:45.135962 16285 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.136595 16285 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.136408 16290 consensus_queue.cc:260] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.137974  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.138720 16285 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.140050 16095 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:45.140690 16127 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:45.141243 16095 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.142010 16095 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:45.140246 16285 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:45.144901 16249 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:45.145998 15705 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.153218 15995 consensus_queue.cc:1048] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:45.156654  8623 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:45.159046 16224 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:04:45.159266  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 132264us
I20260709 13:04:45.159744 16224 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:45.160200  8696 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:45.160686  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.161844 16213 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:45.139011  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 98113.4us
I20260709 13:04:45.163865  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.106815 16300 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:45.119936 16156 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.189119  8655 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:45.193902  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:45.191937  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78613f64484447c487fa71abda1ec1ac" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.197175  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.198495  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.199153  8657 raft_consensus.cc:1275] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:04:45.198338  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:45.183549 16041 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:04:45.227630  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 67453us
W20260709 13:04:45.229672  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 103907us
I20260709 13:04:45.232148  9325 raft_consensus.cc:1240] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:45.156581 16095 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.237655  9296 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.160967 16224 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.141436 16127 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.218833 16290 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:45.204959 16249 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:45.188405 16213 consensus_queue.cc:1048] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 13:04:45.226476 16041 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
W20260709 13:04:45.248914  9294 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:45.243991  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:45.248596  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 110703us
I20260709 13:04:45.252158  8549 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.198091 16300 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.253362  8730 raft_consensus.cc:2368] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 21. Current term is 22.
I20260709 13:04:45.255043  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.258458  8549 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:45.261809  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:45.248806 15995 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:45.237506 16156 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.251233 16041 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:45.271456  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 82561.5us
I20260709 13:04:45.256665 16321 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 13:04:45.272311  8697 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:04:45.266425  8696 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:45.278069  8696 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:45.282485 16321 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:45.283030 16321 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.283443 16321 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:45.289513  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.267208  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:45.270531  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 199696us
I20260709 13:04:45.306922  8732 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:45.260963 15985 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:45.310027  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:45.265087 16295 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:45.335187  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.336725  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.341760  8725 raft_consensus.cc:1275] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:04:45.342639  8728 raft_consensus.cc:2393] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:45.337554  8731 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:04:45.346393  8550 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:45.333354 16342 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 13:04:45.340386 16338 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:04:45.357618  8727 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:45.359294  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:45.340668 16201 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:45.340641 16227 raft_consensus.cc:2804] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:04:45.363591  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 127658us
I20260709 13:04:45.334085 16348 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:45.360605 16347 consensus_queue.cc:1059] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
W20260709 13:04:45.372074  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 110304us
I20260709 13:04:45.373992  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.376215  9296 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:45.378175  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 68218.4us
I20260709 13:04:45.380623  8625 leader_election.cc:304] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:45.360623 16350 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:45.369544 16348 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.359287 16285 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:45.390096  9296 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.391884  8550 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:45.383749 15985 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:45.395841  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 129831us
I20260709 13:04:45.404480  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:45.357168 16249 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.400218 16201 raft_consensus.cc:3060] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:45.393155 16348 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:45.400607 15995 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:45.396742 16227 raft_consensus.cc:697] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:45.424952  8696 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:45.429095  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.394023 16285 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.419143 16095 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:45.427047 16340 log.cc:927] Time spent T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:04:45.436594  8550 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.436331  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:45.445868  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55856.4us
I20260709 13:04:45.432374 16127 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.447727  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.447145  9296 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.451056 16156 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.451373 16317 raft_consensus.cc:2749] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:45.451742 16356 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:45.437903 16300 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.452244 16358 consensus_queue.cc:1059] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 13:04:45.458235 16127 raft_consensus.cc:2804] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:45.443603 15995 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:45.459766  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.445267 16213 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:45.470942 16300 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:45.471706 15538 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:45.511837  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:45.511914  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:45.512908 16367 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.515390 16367 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.457123 16025 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:45.477612 16344 log.cc:927] Time spent T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
W20260709 13:04:45.480433 16345 log.cc:927] Time spent T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:04:45.481163  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.519044 16025 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.449304 16346 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:45.520560 16346 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.520956 16346 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:45.521966  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:45.522673 15985 raft_consensus.cc:3055] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:45.523075 15985 raft_consensus.cc:740] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:45.523952 15985 consensus_queue.cc:260] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.490406 16364 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:45.494191  8728 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 13:04:45.494335 16332 log.cc:927] Time spent T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.560s	user 0.009s	sys 0.000s
W20260709 13:04:45.494846 16343 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 13:04:45.495510 16041 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.496483  8725 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 13:04:45.496975 16349 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.054s	user 0.000s	sys 0.002s
I20260709 13:04:45.556552  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.499754 16285 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.499667 16365 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 13:04:45.479056 16363 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.500047 16266 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.500761 16317 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:45.582569  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 146306us
I20260709 13:04:45.463681 16127 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:45.506449 16341 log.cc:927] Time spent T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: Append to log took a long time: real 0.096s	user 0.000s	sys 0.002s
I20260709 13:04:45.508291  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.459403 16156 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:45.481988 16227 consensus_queue.cc:237] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.525171 15985 raft_consensus.cc:3060] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:45.542961  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 165236us
I20260709 13:04:45.594558 16317 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:45.604935  8469 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 15 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:45.548573  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.550909 16321 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.557775  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:45.558434  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.558861 15538 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:45.543934 16356 consensus_queue.cc:1048] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 13:04:45.547955  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:45.620751  9324 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:45.560916  8724 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 15.
I20260709 13:04:45.497779 16358 raft_consensus.cc:3055] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:45.577816  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.596150  8625 leader_election.cc:400] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 21. Current term is 22.
I20260709 13:04:45.597707 16182 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:45.607177  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:04:45.613895  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 209520us
I20260709 13:04:45.614426  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.708091  8731 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:45.616222  8657 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:45.708402  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.595585 16127 raft_consensus.cc:3060] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:45.667410  9328 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:45.583312 16363 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.645071  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:45.578261 16300 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.092s
I20260709 13:04:45.658114 16041 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.720084  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:45.720909  8650 raft_consensus.cc:2393] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:45.651607 16317 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.652566 16156 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.722445 16156 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.725768 16156 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.662494  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:45.657563  8653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38088 (request call id 3235) took 1257 ms. Trace:
I20260709 13:04:45.684252  9324 raft_consensus.cc:2468] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 22.
I20260709 13:04:45.655505 16266 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.567625 16364 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:45.705511  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 170309us
I20260709 13:04:45.618103  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.594383 16134 consensus_queue.cc:1048] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 13:04:45.732483 16321 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:45.709790  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.711054 16358 raft_consensus.cc:740] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:45.734659  8549 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:45.737748 16366 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:45.738078 16056 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.738680 16366 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:45.738862  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 67923us
I20260709 13:04:45.740338  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:45.742561 16186 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.689s	user 0.010s	sys 0.003s
I20260709 13:04:45.702422 16285 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:45.744592 16156 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:45.744488 16285 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.621263 16025 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.729988  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.669239 16201 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:45.717096  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:45.775501  9331 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 20: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:45.752692  8697 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:45.761125  9326 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:45.703334 16367 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.726696  8653 rpcz_store.cc:276] 0709 13:04:44.400835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.407005 (+  6170us) service_pool.cc:224] Handling call
0709 13:04:44.438951 (+ 31946us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:44.529636 (+ 90685us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:44.529690 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:44.530496 (+   806us) log.cc:844] Serialized 26 byte log entry
0709 13:04:44.745432 (+214936us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:44.745471 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:44.745546 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:45.463383 (+717837us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:45.657210 (+193827us) raft_consensus.cc:1685] finished
0709 13:04:45.657235 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:45.657488 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1287,"raft.queue_time_us":5789,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":28412,"spinlock_wait_cycles":25983872,"thread_start_us":214283,"threads_started":1,"wal-append.queue_time_us":402999}
I20260709 13:04:45.790144 15792 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:04:45.738708 16056 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.739542 16366 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.795595 16056 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.741418 16266 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:45.787262  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:45.795238 16366 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.746430 16249 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:45.803247  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.807574  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:45.811635  8732 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:45.818645  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.832232  9328 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:45.816998 16321 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.786448 16025 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.829206 16201 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:45.896613  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 74055us
I20260709 13:04:45.826862 16290 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.905591  8733 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:45.786414 16364 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.738269 16358 consensus_queue.cc:260] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.809717 16298 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.092s
I20260709 13:04:45.826570 16266 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.923156 16240 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.923775 16240 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.924942 16266 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.926548 16366 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.928339 16025 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.910187 16201 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.932570 16201 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.932422 16346 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.910471 16290 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:45.934136 16285 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:45.935690 16290 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.903046 16362 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:45.900990 16352 raft_consensus.cc:2804] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:45.940172  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.940385  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.946558 16373 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260709 13:04:45.945083 16352 raft_consensus.cc:697] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:45.949121 16352 consensus_queue.cc:237] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.943779 16362 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:45.959342 16362 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.926585 16363 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.962020  8731 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:45.909883 16317 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:45.903401 16321 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:45.974048  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:45.964927 16358 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:45.909946 16348 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:46.034646 16364 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.047379 16321 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:45.974447 16317 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:46.051043  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:46.051717  9326 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:46.055878 16240 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.063174  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:46.061318 16127 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:46.073026 16127 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:46.068631 16346 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:46.074613 16317 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:46.081558  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.085294 16348 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:46.086876  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.088737  8653 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:46.088971  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.094105  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.163681  8658 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:46.163511  8651 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:46.213822  8733 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:46.232997  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.234318  8656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.231997  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.236934  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.231310  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.244508  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:46.245330  8655 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.235749  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:46.238302  8652 raft_consensus.cc:2468] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:46.236253  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.253731 16375 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.254174 16182 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.254503  8654 raft_consensus.cc:1240] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:46.254770 16182 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.257081 16182 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:46.258584 16127 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.230089 16317 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.259186 16127 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.259503 16317 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.262207  9331 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:46.262385  8655 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:46.236361 16346 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.264278  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.267163  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.265609  8731 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 19.
I20260709 13:04:46.272038  8653 raft_consensus.cc:2393] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 23: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:46.272066  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.275856  9324 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:46.276588  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.277307  8659 raft_consensus.cc:2468] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:46.288928  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:46.264935 16356 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:46.295040  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.268000 16127 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:46.280345 16237 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.013s	sys 0.009s
W20260709 13:04:46.281291  8550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.000s	sys 0.010s
I20260709 13:04:46.302285  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.281795 15705 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.311259  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
W20260709 13:04:46.302285  8550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.000s	sys 0.010s
I20260709 13:04:46.315584  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.296221 16356 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.306181  8548 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:46.302141 16237 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.013s	sys 0.009s
I20260709 13:04:46.264636 16346 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.335511  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:46.332803  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.324195  8651 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
W20260709 13:04:46.317252  8696 leader_election.cc:336] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.325816  7481 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 8609)
I20260709 13:04:46.326057  8698 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:46.303059 16347 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:46.339872  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:46.331569  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.345208  8659 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.316109 16134 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.342381  8698 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:46.342372  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.344004  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.340039  8696 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:46.331337 16227 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.018s	sys 0.017s
I20260709 13:04:46.348551 16347 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.354097 16347 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:46.351848 16354 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.085s	sys 0.194s
W20260709 13:04:46.355437 16354 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.086s	sys 0.195s
I20260709 13:04:46.348719 16189 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 13:04:46.351930  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.333041 16356 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:46.357676  8729 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 18.
I20260709 13:04:46.254298 16375 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:46.339525  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 124616us
I20260709 13:04:46.363559  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.363687  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:46.366469 16389 consensus_queue.cc:1059] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:46.368883 16389 raft_consensus.cc:3055] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:46.357987 16134 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.370410 16134 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.369264 16389 raft_consensus.cc:740] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:46.316303 15705 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.371721 16389 consensus_queue.cc:260] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.373160 16389 raft_consensus.cc:3060] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:46.373648 16317 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.374164  8659 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:46.374253 15705 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.360786 16391 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:46.360672 16227 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.018s	sys 0.017s
I20260709 13:04:46.376332  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:46.376608  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:46.380980  9328 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:46.383590  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:46.341288 16348 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:46.384042  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:46.385365 16348 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:46.385478 16056 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:46.385648 15705 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:46.384467 16041 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.386126 15705 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.386090 16056 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.386777 15705 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:46.386607 16041 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.388942 16041 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.389959  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:46.390539  8727 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.392572  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:46.392999 16189 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.393747 16189 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:46.394163 16189 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.395715  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf63362bb9b4409299528319b7472b04" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.396910  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.386272  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:46.397835  8724 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:04:46.384776 16375 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:46.396978  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:46.398838  8547 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.399111  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:46.393352  8654 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 20: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:46.400614  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.399672  8659 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:46.405953  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.408248  8651 raft_consensus.cc:1275] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:46.426849  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:46.427589  9324 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 17.
I20260709 13:04:46.430797  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:46.431762  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 49566.3us
I20260709 13:04:46.422528  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:46.438382  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.445057  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.445977  8731 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
W20260709 13:04:46.459687  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 153669us
I20260709 13:04:46.459724  8625 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:46.460724  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.461802  8550 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
W20260709 13:04:46.462637  8622 leader_election.cc:336] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.466249  9294 leader_election.cc:336] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.361132 16299 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 13:04:46.473032  8622 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:46.475412  8696 leader_election.cc:336] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.437198 16127 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:46.463935 16366 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.481940 16346 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.488921  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.494124  8696 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:46.494537  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.454825 16362 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:46.504307  8731 raft_consensus.cc:1240] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:46.472198 16240 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:46.471752 16364 raft_consensus.cc:2804] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:46.386335 16348 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:46.511423  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:46.496982 16362 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.525434  8727 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:46.507144 16358 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:46.529989  8549 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:46.528748 16388 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.514186 16364 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.509478 16363 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.532466 16351 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:46.528893 16362 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:46.525784  8659 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:46.526607 16350 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:46.496089 16366 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:46.537953 16346 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:46.543131 16385 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:46.545322  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:46.539764  9294 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.553267  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 92696.4us
I20260709 13:04:46.556371  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.533596 16388 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:46.535485 16364 raft_consensus.cc:3060] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:46.537494 16363 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.558615  8549 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:46.560528  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 122507us
I20260709 13:04:46.567111  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:46.561105 16346 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:46.566507 16358 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:46.570268 16346 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.560379 16388 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.569268 16127 raft_consensus.cc:2804] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.567874 16025 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:46.560540 16399 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:46.580003 16358 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.581562  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.593323  8724 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:46.560423 16363 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.599864  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:46.600597  8622 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.618322  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.619673 16385 raft_consensus.cc:3060] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:46.622786 16366 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:46.594743 16189 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.625051 16189 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.594373 16358 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.626859  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.587888 16127 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.634060  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.593943 16240 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:46.647331  8549 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:46.589291 16056 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.650991 16399 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:46.654664 16025 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:46.655369 16025 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
W20260709 13:04:46.656456 16397 log.cc:927] Time spent T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:04:46.656477 16025 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:46.661918 16382 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.119s	user 0.003s	sys 0.000s
I20260709 13:04:46.676584  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.681881  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:46.587427 16356 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.631866 16366 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:46.694653  8623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.646061 16127 raft_consensus.cc:3060] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.642282 16375 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:46.649711 16240 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:46.702661  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 218272us
I20260709 13:04:46.704417  9296 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.709026  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:46.722177  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 121589us
W20260709 13:04:46.724021  8622 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.710191  8550 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:46.725317  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.726158  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:46.726569  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
W20260709 13:04:46.736285  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 182002us
W20260709 13:04:46.737247  8696 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.677255 16352 consensus_queue.cc:1048] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 13:04:46.729031 16365 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:46.692726 16364 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:46.729288 16156 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:46.784374  8549 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:46.726914 16299 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 13:04:46.735958 16213 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:46.762387 16182 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:46.699070 15705 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:46.805519  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 81517us
W20260709 13:04:46.806568  8622 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.810940  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 63583.8us
W20260709 13:04:46.816713  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 79508.7us
W20260709 13:04:46.822188  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:46.829634  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 92501us
W20260709 13:04:46.842760 16405 log.cc:927] Time spent T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.305s	user 0.001s	sys 0.001s
I20260709 13:04:46.844913  8727 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:46.845561  8725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40420 (request call id 3215) took 1039 ms. Trace:
I20260709 13:04:46.845983  8725 rpcz_store.cc:276] 0709 13:04:45.806073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:45.806216 (+   143us) service_pool.cc:224] Handling call
0709 13:04:45.806616 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:46.247985 (+441369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:46.248024 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:46.279965 (+ 31941us) log.cc:844] Serialized 26 byte log entry
0709 13:04:46.348540 (+ 68575us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:04:46.366244 (+ 17704us) log.cc:844] Serialized 14 byte log entry
0709 13:04:46.384208 (+ 17964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:46.384286 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:46.842092 (+457806us) spinlock_profiling.cc:243] Waited 7.32 ms on lock 0x7b4400022b4c. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406a4fc85 00007f1406a506a2 00007f14097c21db 00007f14097bdaaf 00007f14097b186e 00007f14097b0bb6 00007f14097b07d8 00007f14097ba221 00007f14097ba1d9 00007f14097ba169
0709 13:04:46.845226 (+  3134us) raft_consensus.cc:1685] finished
0709 13:04:46.845333 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:46.845500 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33533,"prepare.queue_time_us":15605,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":38,"raft.queue_time_us":57459,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":40024,"spinlock_wait_cycles":16833408,"thread_start_us":443287,"threads_started":2,"wal-append.queue_time_us":181653}
I20260709 13:04:46.851790  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.855145  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.858613  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:46.867544 16224 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:46.868214 16224 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.879356  8652 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260709 13:04:46.879901  8652 pending_rounds.cc:85] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 14
W20260709 13:04:46.894215  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:46.828979 16366 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:46.809836 16213 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:46.917258 16213 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:46.880172  8652 pending_rounds.cc:107] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 13:04:46.875847 16025 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.908478 15705 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.910421 16358 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.832196 16365 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.928797  8652 raft_consensus.cc:2889] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:46.933429  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:46.936259 16240 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:46.945611  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 139077us
W20260709 13:04:46.957136  8622 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.925814  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.957981  9294 leader_election.cc:336] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.958313  8696 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:46.948441  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 164145us
W20260709 13:04:46.926409 16134 consensus_meta.cc:287] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: Time spent flushing consensus metadata: real 0.556s	user 0.009s	sys 0.004s
I20260709 13:04:46.968254  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:46.969033  8651 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:46.973444 16423 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:46.948328 16025 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.975181 16240 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:46.976176 16025 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:46.980132 16415 log.cc:927] Time spent T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 13:04:46.961896 16365 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:46.984045 16365 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:46.984830 16041 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:46.988608  8651 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Rejecting Update request from peer 1dc41db601fe4b91aa6d461b504da0e7 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:46.962327 16347 consensus_meta.cc:287] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.608s	user 0.012s	sys 0.000s
I20260709 13:04:47.015305  8659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:47.017557  8659 raft_consensus.cc:2393] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 20: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:47.036026  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:47.034852 16423 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:47.041298 16240 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:47.041764 16240 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:47.029618 16358 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 13:04:47.042670 16240 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.044271 16240 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:47.045321 16364 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.050001 16356 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:47.052063 16418 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 13:04:47.054382 16420 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 13:04:47.054574  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.056335  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:47.057830 16413 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.148s	user 0.002s	sys 0.000s
I20260709 13:04:47.059211 16425 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:47.061029  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 103934us
W20260709 13:04:47.062053  8622 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:47.063344  8622 leader_election.cc:336] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.063678  8622 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:47.065256  8476 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 18 to 21, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 21 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.066139 16385 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:46.960230 16362 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.066920  8476 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 19 to 22, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 22 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.068751 16362 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:47.068284  8476 catalog_manager.cc:5697] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 16, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 16 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.030591 16388 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.070268  8476 catalog_manager.cc:5697] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 17, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 17 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:47.072547  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 146756us
W20260709 13:04:47.074203  8696 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.061232  8550 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:47.082448  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.082715  8724 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:04:47.084633  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:47.085350  8733 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:47.096187  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:47.104035  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.081347 16385 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:47.107847 16358 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:47.110170 16224 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:47.120040  8696 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:47.188725  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 68753.9us
W20260709 13:04:47.189855  8696 leader_election.cc:336] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:47.191434  8696 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.191933  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.123658 16358 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.193748 16358 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.196467 16358 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:47.125908  8733 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:47.128283  8550 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:47.198211  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 70017.5us
I20260709 13:04:47.199352  8550 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.200575 16025 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:47.071653  8476 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 18, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.200959 16025 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:47.201337 16025 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:47.202132 16348 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:47.202821 16348 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:47.204368 16348 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.204754 16348 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:47.205922  8476 catalog_manager.cc:5697] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 17, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.207901 16373 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:47.207369  8476 catalog_manager.cc:5697] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:47.085948  9294 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:47.209813  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 123904us
I20260709 13:04:47.169468  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:47.213127  8726 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:47.172626  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
W20260709 13:04:47.174250 16414 log.cc:927] Time spent T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.261s	user 0.000s	sys 0.002s
W20260709 13:04:47.176364  8475 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 309) took 72462 ms (client timeout 15000 ms). Trace:
I20260709 13:04:47.148550 16411 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:47.215739  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:47.186898 16417 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.272s	user 0.002s	sys 0.000s
W20260709 13:04:47.187624 16419 log.cc:927] Time spent T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.274s	user 0.002s	sys 0.000s
I20260709 13:04:47.208966  8476 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 16. New cstate: current_term: 16 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:47.214815  8475 rpcz_store.cc:269] 0709 13:03:34.714036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:34.714281 (+   245us) service_pool.cc:224] Handling call
0709 13:04:47.176213 (+72461932us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:44.637928 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:44.638631 (+   703us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:44.638677 (+    46us) write_op.cc:435] Acquired schema lock
0709 13:04:44.638708 (+    31us) tablet.cc:662] Decoding operations
0709 13:04:44.639509 (+   801us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:44.639598 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:44.639623 (+    25us) tablet.cc:685] Acquiring locks for 5 operations
0709 13:04:44.640364 (+   741us) tablet.cc:701] Row locks acquired
0709 13:04:44.640408 (+    44us) write_op.cc:260] PREPARE: finished
0709 13:04:44.640603 (+   195us) write_op.cc:270] Start()
0709 13:04:44.640759 (+   156us) write_op.cc:276] Timestamp: P: 1783602284640525 usec, L: 0
0709 13:04:44.640868 (+   109us) op_driver.cc:348] REPLICATION: starting
0709 13:04:44.717576 (+ 76708us) log.cc:844] Serialized 2059 byte log entry
0709 13:04:46.365949 (+1648373us) op_driver.cc:464] REPLICATION: finished
0709 13:04:46.570120 (+204171us) write_op.cc:301] APPLY: starting
0709 13:04:46.570258 (+   138us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:46.588950 (+ 18692us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:46.589070 (+   120us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:46.592227 (+  3157us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:46.592535 (+   308us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=10,mrs_lookups=0
0709 13:04:46.592595 (+    60us) write_op.cc:312] APPLY: finished
0709 13:04:46.592890 (+   295us) log.cc:844] Serialized 57 byte log entry
0709 13:04:46.594513 (+  1623us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:46.594621 (+   108us) write_op.cc:454] Released schema lock
0709 13:04:46.594696 (+    75us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:46.617297 (+ 22601us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:46.625483 (+  8186us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":178977,"apply.run_cpu_time_us":9355,"apply.run_wall_time_us":45647,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25017,"num_ops":5,"prepare.queue_time_us":454797,"prepare.run_cpu_time_us":169590,"prepare.run_wall_time_us":639461,"raft.queue_time_us":571126,"raft.run_cpu_time_us":55239,"raft.run_wall_time_us":201683,"replication_time_us":1705568,"thread_start_us":1229509,"threads_started":4,"wal-append.queue_time_us":605351}]]}
I20260709 13:04:47.213632 16348 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.216354  8729 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 19.
I20260709 13:04:47.222254  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:47.220795  8476 catalog_manager.cc:5697] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:47.228320  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.232373  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.260577  8651 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:47.269891  8729 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:47.271550  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:47.274168  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 210823us
W20260709 13:04:47.275233  8622 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.275905  8622 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:47.227464 16363 consensus_meta.cc:287] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.613s	user 0.010s	sys 0.000s
I20260709 13:04:47.272449  8732 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:47.277949  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:47.282084  8654 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:46.987588 16134 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.287122  8476 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.244015 16385 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:47.258011 16056 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.002s	sys 0.013s
I20260709 13:04:47.294123  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.294775  8731 raft_consensus.cc:3060] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:47.292680  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:47.292017  8724 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:47.298180  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:47.298815  9325 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:47.086637 16347 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.254685 16358 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:04:47.307148  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 78868.1us
W20260709 13:04:47.309480  8696 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:47.311160  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:47.270437 16156 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:47.316891  9328 raft_consensus.cc:1240] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Rejecting Update request from peer a0d703ee18464475a290425976246555 for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:47.300626 16056 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.002s	sys 0.013s
I20260709 13:04:47.318456  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:47.324209  9331 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 21: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:47.328820  8731 raft_consensus.cc:2468] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
I20260709 13:04:47.313587 16358 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:47.339178  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63981.6us
W20260709 13:04:47.340732  7481 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 8609)
I20260709 13:04:47.286106 16363 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:47.313189 16393 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:47.308151 16282 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:47.309111 16356 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:04:47.372349 16127 consensus_meta.cc:287] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.646s	user 0.005s	sys 0.003s
I20260709 13:04:47.373361 16134 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.376056  9325 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:47.376693 16156 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:47.378706 16347 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:47.381738  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 55652.3us
W20260709 13:04:47.394912  8622 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.360445  8476 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 17, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 17 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.306046 16385 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:47.414618  9296 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.014s	sys 0.031s
W20260709 13:04:47.428918  9296 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.015s	sys 0.032s
W20260709 13:04:47.429248  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 607525us
I20260709 13:04:47.432605  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:47.433296  8651 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
I20260709 13:04:47.395439 16358 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:47.435166  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 47678.4us
W20260709 13:04:47.431730 16432 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260709 13:04:47.431464 16426 log.cc:927] Time spent T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.094s	user 0.000s	sys 0.003s
I20260709 13:04:47.452405  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:47.451406  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:47.446779  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:47.454711  9331 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
W20260709 13:04:47.449410  9294 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.428622 16127 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:47.417229 16356 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.458595  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:47.461131  9331 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 21: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:47.376107 16025 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.463901  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:47.464311 16025 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:47.467608 16388 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:47.469380  8476 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 11 to 15, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 15 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:47.471552  8622 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.479102  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:47.472328  8622 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:47.488296  9327 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:47.496107 16363 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.497970  9327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:47.502249  8476 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 21. New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.504031 15995 consensus_queue.cc:1059] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:47.504294  8476 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.505167 15995 raft_consensus.cc:3055] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:47.508239 15995 raft_consensus.cc:740] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:47.509323 15995 consensus_queue.cc:260] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.510725 16385 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:47.511565  8623 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:47.519032  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:47.521534  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.521741  9297 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:47.523118  9328 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43640 (request call id 1668) took 1134 ms. Trace:
I20260709 13:04:47.523480  9328 rpcz_store.cc:276] 0709 13:04:46.382056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:46.382233 (+   177us) service_pool.cc:224] Handling call
0709 13:04:46.435474 (+ 53241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:47.317341 (+881867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:47.516204 (+198863us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b5400693054. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:04:47.516356 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2026610432}
I20260709 13:04:47.526712  9327 raft_consensus.cc:1275] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 13:04:47.516885 16408 log.cc:927] Time spent T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:04:47.528064  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:47.530591  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.491531 16347 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.520529 16388 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:47.536154  8651 raft_consensus.cc:1240] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:47.522768 16351 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:47.538254  8476 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.536067  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:47.531944 16434 log.cc:927] Time spent T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 13:04:47.533789 16393 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:47.537313 16425 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:47.546355 16282 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:47.532886 16363 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.544094  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.547252 16363 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:47.547853 16435 log.cc:927] Time spent T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:04:47.548071  8724 raft_consensus.cc:2393] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 20: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:47.511269 15995 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:47.534548 16368 raft_consensus.cc:2804] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:47.554737 16368 raft_consensus.cc:697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:47.555723 16368 consensus_queue.cc:237] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.557533 16127 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.549683 16189 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:47.550671  8697 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.562384 16356 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:47.539522 16428 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.010s
I20260709 13:04:47.563203 16421 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.494971 16025 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:47.567198  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 106683us
I20260709 13:04:47.536018  9296 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:47.573784  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.525543 16295 consensus_queue.cc:1059] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 13:04:47.570737  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:47.614995  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.619212  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.620059  8725 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:04:47.622628  8727 raft_consensus.cc:2393] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 21: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:47.568740 16351 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:47.578591 16362 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:47.540469 16347 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:47.630939  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 95000us
I20260709 13:04:47.578370 16364 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 13:04:47.634475 16134 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:47.635008 16134 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:47.635426 16134 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:47.635851 16321 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.636409 16321 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.637724  8726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.638340  8726 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:04:47.645073  9324 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:47.646653  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:47.647987  9328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:47.655092  8476 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 18, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 18 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:47.657169  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 114518us
I20260709 13:04:47.626961 16127 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:47.630597 16439 log.cc:927] Time spent T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:04:47.673709 16385 consensus_queue.cc:1059] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:47.675483  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:47.676929 16424 raft_consensus.cc:493] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.627650 16189 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:47.700371  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:47.599182 16025 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:47.705174  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:47.595515 16421 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.709776  8476 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 19, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 19 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.652266 16134 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.729364  8726 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:47.680126 16425 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
W20260709 13:04:47.742707  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 84038us
W20260709 13:04:47.744037  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 66675.8us
I20260709 13:04:47.702214 16362 consensus_queue.cc:1048] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 13:04:47.749378  9324 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 22: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:47.720669 15995 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:04:47.851836  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:47.870743  8622 leader_election.cc:336] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.871184  8622 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:47.866194 16453 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.746424  8476 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 18, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.708873 16424 raft_consensus.cc:515] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.814795 16356 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:47.907527  9330 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:47.909708  8725 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 22.
I20260709 13:04:47.911788  8476 catalog_manager.cc:5697] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.943292  8550 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:47.946303  8476 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 17, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 17 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:47.950397  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 204638us
W20260709 13:04:47.907975 16417 log.cc:927] Time spent T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:04:47.933679 16387 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:47.881021 16321 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.902307 16425 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.887189 16134 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.979789  8733 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:04:47.959126 16449 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:47.992297 16025 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:47.955241  8476 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 16 to 18, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 18 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:47.873646 16371 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:47.941015 16356 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:48.007658  8696 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.000341  8476 catalog_manager.cc:5697] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 20, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 20 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:48.008322  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.012398 16424 leader_election.cc:290] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.012846 16428 raft_consensus.cc:3055] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:48.013288 16428 raft_consensus.cc:740] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:48.014389 16421 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.013324 16189 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:48.020144  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 82395.8us
W20260709 13:04:48.021998  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 537644us
I20260709 13:04:48.027495 16356 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:48.041742  8476 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 21. New cstate: current_term: 21 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:48.081841  8476 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 11 to 12, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 12 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:48.083254  8623 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7
I20260709 13:04:47.978119 16425 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:48.079882 16366 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.747321 16351 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:47.907581 16453 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.119154 16363 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:48.114157  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.114897  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:48.126520  8652 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:04:48.090781 16025 raft_consensus.cc:493] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:48.131883 16025 raft_consensus.cc:515] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.134905  8652 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 22.
I20260709 13:04:48.135160 16025 leader_election.cc:290] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.005666 16371 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.137735 16425 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:48.139163 16371 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:48.116115 16366 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.140678 16300 raft_consensus.cc:3055] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:48.141037 16300 raft_consensus.cc:740] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:48.114675  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.014310 16428 consensus_queue.cc:260] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:48.143436 16358 consensus_meta.cc:287] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Time spent flushing consensus metadata: real 0.700s	user 0.007s	sys 0.003s
I20260709 13:04:48.144018 16358 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:48.119858  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.092159 15705 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:04:48.151486  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 141243us
W20260709 13:04:48.152457  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:48.154080  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:48.155826  8659 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:48.156157  8658 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:48.154742  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:48.153563  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:48.157428  8654 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 22: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:48.155229  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.157621  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.107033 16189 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:48.166185  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.173084  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.175310  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.149978 16366 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.168080 15705 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
W20260709 13:04:48.186942  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 316212us
W20260709 13:04:48.187811  8622 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.188207  8622 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:48.170095 16453 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.198863  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:48.204156  8622 leader_election.cc:336] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.205452  8622 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:48.207345 15705 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.211373 16449 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.211953 16449 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.214133 16449 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.194193 16300 consensus_queue.cc:260] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.214958 16300 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:48.203908 16237 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:48.219878 16134 consensus_queue.cc:1175] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d0a2b9c938d94a4fa258e068ea054d84" responder_term: 21 status { last_received { term: 21 index: 16 } last_committed_idx: 16 last_received_current_leader { term: 21 index: 16 } }
I20260709 13:04:48.222213 16358 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:48.191217 16421 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 13:04:48.201601 16366 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:48.249010 16347 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.254436  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f9dd5b88e0647f284658784381e1893" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.258585  8658 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:48.254953  9329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.262854  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.243654 16428 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:48.269747  8548 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.271284  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.272305  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.273203  8550 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.254487  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.267891  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.295990 16465 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:48.296895 16465 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.297598 16453 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:04:48.301965  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 97829.4us
I20260709 13:04:48.302551 16237 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:48.302752  8622 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.303251  8622 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:48.304350  8622 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:48.305294  8622 leader_election.cc:336] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.305647  8550 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:48.305693  8622 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:48.307128  8550 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:48.320593  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "689ad8326d634dee8402f1352a76f0e2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.320758  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:48.322331  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 69479.8us
I20260709 13:04:48.312126 16366 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:48.335055  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4f31f7ca2bf4416bec3531998325cc4" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.336349  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.337011  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.338212  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:48.338624 16041 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:48.338935  8727 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 22: Already voted for candidate a0d703ee18464475a290425976246555 in this term.
I20260709 13:04:48.339180 16041 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.339742  8727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.340423  8727 raft_consensus.cc:2393] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 17: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:48.342995  8730 raft_consensus.cc:1240] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Rejecting Update request from peer 453e644bad5d410ab9be85b353e28dc2 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:48.345999  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.346299  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0337bfcbeb3b4e15b96669c8d3ca4692" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.346596  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.350764  8550 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.345691  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.328104  8733 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:48.365665  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.371212  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.362004  8548 leader_election.cc:304] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:48.387001  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:48.341380 16041 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:48.388409  7481 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 8609)
I20260709 13:04:48.389127 16421 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:48.389322  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:48.390041 16421 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:48.390210  8725 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:48.392375  8548 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:48.393231 16358 raft_consensus.cc:2749] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:48.394018 16358 raft_consensus.cc:3060] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:48.395699  9294 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.396243  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.318074 16363 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:04:48.398914  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 195229us
W20260709 13:04:48.399590  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 94317.7us
I20260709 13:04:48.401842  8730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.318080 16444 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:48.326046 16025 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.325649 16347 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:48.407836  8622 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:48.409907  9294 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.410341  8622 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:48.410678  9294 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:48.415930  9294 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.416429  9294 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.411460 16363 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:48.411154 16356 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:04:48.422394  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 154578us
I20260709 13:04:48.417824  8550 leader_election.cc:304] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:48.429339  8622 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:48.436038  8550 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:48.436873 16365 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:48.437599 16282 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:48.441035  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.419531 16182 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:48.413549 16425 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 13:04:48.443928  8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:48.444532  8658 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:48.409461 15705 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:48.413319 16237 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:04:48.450728 16363 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:48.412261 16351 raft_consensus.cc:3055] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:04:48.454728  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:48.456624  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.462704  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.464582  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd0fc940d95d4b118d3e2564ed48b09e" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.411895 16025 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:48.454767 16449 raft_consensus.cc:2749] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:48.548355  8728 raft_consensus.cc:1275] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:48.550931  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:48.557180  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 78727.5us
W20260709 13:04:48.564558  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 57115.2us
I20260709 13:04:48.570413  9296 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.472913 16127 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:48.565871  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:48.472975 16351 raft_consensus.cc:740] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:48.584792  8725 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:48.581624 16134 raft_consensus.cc:2749] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:48.571686 16356 consensus_queue.cc:1048] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.010s
W20260709 13:04:48.570554  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 74773.8us
I20260709 13:04:48.579979  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.587754  8550 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:48.552018 16421 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:48.584976  8658 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:48.566157 16041 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.594533 16364 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:48.606272 16430 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:48.622005 16363 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.641337 16449 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:48.643596 16295 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:48.618973 16041 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:48.710181 16474 log.cc:927] Time spent T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:04:48.712265 16471 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:04:48.714125 16025 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:48.699678 16350 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:48.715777 16350 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.723042 16350 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:48.726467  9294 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 124 similar messages]
I20260709 13:04:48.732080 16465 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:48.684000 16449 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:48.740379  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:48.737253  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:48.753695  8729 raft_consensus.cc:1240] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:04:48.675467 16363 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:48.755043  9325 raft_consensus.cc:2393] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 20: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:48.677289 16351 consensus_queue.cc:260] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.771103 16449 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:48.778293  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:48.750603 16393 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:48.791003 16393 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:48.799045 16454 consensus_queue.cc:1059] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:48.790428 16363 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:48.812451  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 224790us
I20260709 13:04:48.821472  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.830096  9324 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:04:48.812901 16450 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:48.869256  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 272806us
I20260709 13:04:48.870267  8549 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.833930 16358 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:48.871089  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.883531  9331 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:48.879638 16393 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:48.873967 16156 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:48.892624  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51607.9us
I20260709 13:04:48.895599  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.867065 16363 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:48.911499  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 69381us
I20260709 13:04:48.938503  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.940936  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:48.846287 16450 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.945186  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.951539  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96cbbc16390646a9875b38f1b174f981" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:48.957507 16041 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:48.957779 16478 log.cc:927] Time spent T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 13:04:48.912482 16363 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.963783 16363 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.966338 16025 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:48.966524  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.966915 16025 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:48.973944  8730 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 13:04:48.974130  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:48.977835  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:48.969137 16025 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:48.942816 16387 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 13:04:48.996196  8728 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:04:48.980645 16041 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:49.006109  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:49.017051  8622 leader_election.cc:336] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.017730  8622 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:49.018656  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.022060  8549 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.009954 16041 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:49.036677  8622 leader_election.cc:336] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.046965  8622 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:49.052281  8549 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:49.055037  8622 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.055711  8622 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:49.007742 16283 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 13:04:49.029582 16351 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:49.064468  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:49.065320  8550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 65314.8us
I20260709 13:04:49.052573 16041 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:49.049049 16459 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:49.081638  8549 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.065228 16388 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:49.087878  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 79281.3us
I20260709 13:04:49.062039 16445 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:49.057582 16366 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:49.066918 16453 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:49.081383 16364 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:49.103751  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:49.106901  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:49.092060 16025 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:49.121256  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.084873 16449 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
W20260709 13:04:49.127281  8696 leader_election.cc:336] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.135841  8696 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
W20260709 13:04:49.139742 16491 log.cc:927] Time spent T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 13:04:49.124765 16450 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.125458 16366 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.147854 16352 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:49.147533 16366 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:49.180167 16366 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:49.180824  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.198573 16366 raft_consensus.cc:493] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:49.199213 16366 raft_consensus.cc:515] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:49.203940  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:49.210043  8657 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:04:49.210631  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.214622 16421 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:49.215221 16421 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.219563  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.221774 16366 leader_election.cc:290] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:49.217325 16421 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.227510 16041 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.231901  8728 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:49.234526 16041 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.252498  8729 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:04:49.253717  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:49.268440  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 57881.2us
I20260709 13:04:49.272065  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:49.272818  9326 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:49.272845  8729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.278990  8550 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:49.271662  8696 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.280985  8696 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:49.261231 16481 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.290982  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.292338  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f057517aa4a1496ab40134759e3d8b54" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.282907 16366 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:49.288574 16425 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:49.314850  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.305234  9326 raft_consensus.cc:2468] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:49.323262  8697 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:49.324437 16041 raft_consensus.cc:2804] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:49.325055 16041 raft_consensus.cc:697] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:49.296041 16350 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:49.295452 16424 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.290269 16421 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 13:04:49.333313  8549 leader_election.cc:304] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.336109 16366 raft_consensus.cc:2749] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:49.294760 16481 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.350394 16481 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.334456 16041 consensus_queue.cc:237] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.354331 16487 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:49.358503  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.362076  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947646ebfc34bd7a5a55312f5812ee2" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:49.366081  9331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.331336 16365 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.378189  8550 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:49.334736 16450 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:49.382485 16156 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:49.385530 16365 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.387540 16481 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
W20260709 13:04:49.393498  7481 thread.cc:527] Waited for 4000ms trying to join with heartbeat (tid 8609)
I20260709 13:04:49.397780  9326 raft_consensus.cc:1240] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Rejecting Update request from peer d0a2b9c938d94a4fa258e068ea054d84 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:49.335171 16424 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.412684  8730 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:04:49.416292 16025 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:49.403748 16450 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:49.434612  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:49.401199 16481 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:49.425802 16458 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:49.447949 16493 log.cc:927] Time spent T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: Append to log took a long time: real 0.319s	user 0.002s	sys 0.001s
I20260709 13:04:49.452960 16365 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:49.455837 16424 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.459824  8725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.469450  8725 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 22.
I20260709 13:04:49.461390  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.467938  8650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.472091  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d18196b7e714cb6a79ec64ea3d65803" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:49.474951  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 187387us
I20260709 13:04:49.486514 16350 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:49.466864 16282 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260709 13:04:49.492213 16299 raft_consensus.cc:3055] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:49.493155 16299 raft_consensus.cc:740] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:49.492375 16224 consensus_queue.cc:1059] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:49.495029 16224 consensus_queue.cc:1243] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Peer 1dc41db601fe4b91aa6d461b504da0e7 log is divergent from this leader: its last log entry 17.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:04:49.429525 16450 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.497181 16450 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.466866 16430 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.500877 16430 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.503715 16430 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:49.484810  9297 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.509081  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.509473  8548 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.511193 16487 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:49.523080  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.524968  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed0f0ed87f74c1fb8320d38bceeac72" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.529197  8655 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
W20260709 13:04:49.531152  8696 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.531844  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.469357 16458 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.506455 16430 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:49.572736 16430 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.573187 16430 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:49.558179  9294 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:49.573937  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 49941.5us
I20260709 13:04:49.565613  8728 raft_consensus.cc:2410] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 13:04:49.578477  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:04:49.580808  8549 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 46203.4us
I20260709 13:04:49.584962  8549 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.585727  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:49.549649 16481 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:49.591820 16449 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 13:04:49.595849 16299 consensus_queue.cc:260] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.597295 16299 raft_consensus.cc:3060] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:49.601104 16449 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:49.613832  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:49.613991  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 82904.4us
I20260709 13:04:49.615684  8657 raft_consensus.cc:2410] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 13:04:49.607237 16487 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.620352 16456 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:49.621798 16458 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.629667 16356 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:49.634229  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.635684  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.651681  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.654047 16352 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.654625 16352 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.661207  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.664848 16428 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:49.667510 16430 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.669732 16430 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:49.670362  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:49.656498 16227 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:49.671157  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:49.671752  8733 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:49.639444 16487 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.687842  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.688602  8728 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:49.689286  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.689949  9324 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 20.
I20260709 13:04:49.689854  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:49.695353 16352 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.697237 16454 raft_consensus.cc:493] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 453e644bad5d410ab9be85b353e28dc2)
I20260709 13:04:49.698068 16025 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:49.697831 16454 raft_consensus.cc:515] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.698524 16025 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.699134 16025 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:49.700004 16454 leader_election.cc:290] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.701813  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:49.702440  8653 raft_consensus.cc:3060] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:04:49.703687  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.685206 16487 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.716969  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.717653  9330 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 22.
I20260709 13:04:49.712561  8653 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 23.
I20260709 13:04:49.713654  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77d6200607074a5ebd3e4ae470a2d6ed" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.714838  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.722939  8733 raft_consensus.cc:2468] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 23.
I20260709 13:04:49.723091  8655 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 22.
I20260709 13:04:49.729964  8698 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:49.730893  9324 raft_consensus.cc:2368] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 22. Current term is 23.
I20260709 13:04:49.723313  9296 leader_election.cc:304] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:49.736532 16456 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:49.737005 16456 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.737363 16456 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:49.749883  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:49.757612 16300 raft_consensus.cc:2804] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:49.760015 16300 raft_consensus.cc:697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:49.757457  8623 leader_election.cc:400] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 22. Current term is 23.
I20260709 13:04:49.762511  8623 leader_election.cc:403] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:04:49.763423  8469 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:49.765796 16300 consensus_queue.cc:237] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.777256 16025 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.780138 16456 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.781814 16025 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.766741 16454 raft_consensus.cc:2749] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133) with higher term. Message: Invalid argument: T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 for earlier term 22. Current term is 23.
I20260709 13:04:49.784757  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:49.785373  9330 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:49.787415  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:49.793074  9330 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:49.795280  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:49.795938  8731 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:49.798146  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:49.799145  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:49.805256  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:49.805756  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48906.6us
W20260709 13:04:49.806314  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:49.809860  8731 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d703ee18464475a290425976246555 in term 21.
I20260709 13:04:49.818586  8550 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555; no voters: 
I20260709 13:04:49.824047 16481 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:49.826713 16481 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 21 LEADER]: Becoming Leader. State: Replica: a0d703ee18464475a290425976246555, State: Running, Role: LEADER
I20260709 13:04:49.827596 16481 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:49.841240  8696 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.849648  8657 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:49.879253 16299 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.879958 16299 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.847960 16487 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.882205 16487 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:49.883409 16492 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:49.883715  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.884083 16492 consensus_queue.cc:1243] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Peer d0a2b9c938d94a4fa258e068ea054d84 log is divergent from this leader: its last log entry 18.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:04:49.884371  8731 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 18.
I20260709 13:04:49.884635 16487 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:49.885917  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:49.888590 16454 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:49.889079 16454 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.889523 16454 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:49.890574  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.892809  8657 pending_rounds.cc:85] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Aborting all ops after (but not including) 14
I20260709 13:04:49.893086  8657 pending_rounds.cc:107] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 13:04:49.893324  8657 raft_consensus.cc:2889] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:49.897421 16456 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.899369 16489 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.899881 16489 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:49.900969  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "294596405ca140769bf30fa7ee1e4820" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.901947 16489 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:49.899139  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:49.902732  8550 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.903839 16489 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:49.902910  8654 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:49.908807  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:49.909446  9324 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:49.913364  8654 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 23.
I20260709 13:04:49.914659  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:49.916277 16299 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.917827  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:49.920871  9324 raft_consensus.cc:2468] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 23.
I20260709 13:04:49.929761  8697 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:49.922212 16454 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:49.924461  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeef0c4282c74b3692ab849adb5ce58e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:49.936793 16352 raft_consensus.cc:2804] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:49.936589  8549 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:49.937543 16352 raft_consensus.cc:697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:04:49.925014  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:49.933172  9324 raft_consensus.cc:1275] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:49.939811 16487 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:49.944662 16492 consensus_queue.cc:1048] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:49.940894 16352 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:49.958107  8696 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.963347  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:49.964051  8731 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:49.973126 16454 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Requested vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:49.974144  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:49.975876  8731 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:49.982494  8625 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:49.987618 16299 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:49.988268 16299 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:49.989187 16299 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:50.007941 16487 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.007834 16481 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.011586 16487 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:50.050465  8476 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 208) took 15790 ms (client timeout 15000 ms). Trace:
I20260709 13:04:50.051995 16481 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:50.054318 16481 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:50.051133  8476 rpcz_store.cc:269] 0709 13:04:34.255801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:42.594029 (+8338228us) service_pool.cc:224] Handling call
0709 13:04:50.046347 (+7452318us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:48.329487 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:48.330060 (+   573us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:48.330095 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:04:48.330113 (+    18us) tablet.cc:662] Decoding operations
0709 13:04:48.445810 (+115697us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:48.445906 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:48.445944 (+    38us) tablet.cc:685] Acquiring locks for 24 operations
0709 13:04:48.448137 (+  2193us) tablet.cc:701] Row locks acquired
0709 13:04:48.448162 (+    25us) write_op.cc:260] PREPARE: finished
0709 13:04:48.448326 (+   164us) write_op.cc:270] Start()
0709 13:04:48.448459 (+   133us) write_op.cc:276] Timestamp: P: 1783602288448273 usec, L: 0
0709 13:04:48.448483 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 13:04:48.449257 (+   774us) log.cc:844] Serialized 9316 byte log entry
0709 13:04:49.148554 (+699297us) op_driver.cc:464] REPLICATION: finished
0709 13:04:49.294042 (+145488us) write_op.cc:301] APPLY: starting
0709 13:04:49.303529 (+  9487us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:49.497804 (+194275us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:49.497901 (+    97us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:49.782438 (+284537us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:49.782785 (+   347us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=48,mrs_lookups=0
0709 13:04:49.782841 (+    56us) write_op.cc:312] APPLY: finished
0709 13:04:49.783263 (+   422us) log.cc:844] Serialized 212 byte log entry
0709 13:04:49.783445 (+   182us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:49.783638 (+   193us) write_op.cc:454] Released schema lock
0709 13:04:49.783702 (+    64us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:49.811416 (+ 27714us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:49.811453 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12526,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":134243,"apply.run_cpu_time_us":38221,"apply.run_wall_time_us":524738,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104399,"num_ops":24,"prepare.queue_time_us":82842,"prepare.run_cpu_time_us":49737,"prepare.run_wall_time_us":538815,"raft.queue_time_us":134064,"raft.run_cpu_time_us":32430,"raft.run_wall_time_us":116095,"replication_time_us":691906,"thread_start_us":547167,"threads_started":4,"wal-append.queue_time_us":438061}]]}
I20260709 13:04:50.061565  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:50.068482 16487 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:50.078068  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:50.081295  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:50.081974  8474 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 14 to 17, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 17 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.094684  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:50.097584  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.098181  8474 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 6 to 16. New cstate: current_term: 16 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.104058  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7778779e8695473fa6249308e7a4fa1e" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:50.104669  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.101637 16449 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:50.122519  8474 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 13 to 16, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 16 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:50.133706  8549 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:50.138358  8474 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 reported cstate change: term changed from 11 to 17. New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:50.142675  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.143894 16449 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:50.145457  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:50.150555  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:50.158532  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.168821  8474 catalog_manager.cc:5697] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 11 to 18, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 18 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:50.187885  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.193989 16481 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.195005 16481 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:50.202690  8474 catalog_manager.cc:5697] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 14 to 16, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 16 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.208892  8474 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 12 to 17, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 17 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.214031  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:50.216565  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:50.216187  8474 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 7 to 15, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 15 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:50.218010  8474 catalog_manager.cc:5697] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.219600  8474 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 reported cstate change: term changed from 11 to 14, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 14 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.221496 16481 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:50.221571  8474 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 12 to 17, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.223407 16481 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.224779  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:50.224884 16481 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:50.226585  8654 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:04:50.229455  8733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:50.231045 16449 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:50.231812 16481 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:50.235095  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:50.236709  8654 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 23: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:50.236971  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "081470ef13974fe497f88859ada43ffa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:50.237849  9330 raft_consensus.cc:2393] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0d703ee18464475a290425976246555 in current term 23: Already voted for candidate 1dc41db601fe4b91aa6d461b504da0e7 in this term.
I20260709 13:04:50.233112  8474 catalog_manager.cc:5697] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 13 to 16, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 16 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.239001  8548 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:50.240429 16450 raft_consensus.cc:3060] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:50.241752  8474 catalog_manager.cc:5697] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 9 to 15, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.243717  8474 catalog_manager.cc:5697] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 14 to 17, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.245731  8474 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 10 to 15, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 15 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.232662 16465 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 13:04:50.247942  8474 catalog_manager.cc:5697] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 7 to 17, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.261807 16450 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:50.286595  8530 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:50.287075  8530 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:50.287420  8530 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:50.314162  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.328238 16465 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:50.328831 16465 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:50.330415  8733 raft_consensus.cc:1275] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:04:50.331782 16465 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:50.319141 16481 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:50.341858  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.351545  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:50.351704 16430 consensus_queue.cc:1048] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 13:04:50.352463 16481 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:50.358191 16481 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:50.359652  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:50.362025  9324 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:50.364231  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04399390cf414886a3453981dc1f74aa" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:50.368615  8548 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:50.369547 16449 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:50.369997 16364 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:50.373417  8696 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:50.382409  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.383697  8733 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer a0d703ee18464475a290425976246555: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:50.385519 16481 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 13:04:50.393325  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 61562.9us
I20260709 13:04:50.395538  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:50.396673  8731 raft_consensus.cc:2368] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19.
W20260709 13:04:50.398140  7481 thread.cc:527] Waited for 5000ms trying to join with heartbeat (tid 8609)
I20260709 13:04:50.402822  9297 leader_election.cc:400] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19.
I20260709 13:04:50.405025  9297 leader_election.cc:403] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:50.417825 16528 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 19
W20260709 13:04:50.425294  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.429162  8728 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:50.439288 16481 raft_consensus.cc:493] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.442205 16481 raft_consensus.cc:515] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:50.446113  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:50.446113  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c87eef075e49ba98e563e6fcc45d83" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:50.437763 16283 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 13:04:50.450124  8548 leader_election.cc:304] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:50.451020 16364 raft_consensus.cc:2749] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:50.445127 16481 leader_election.cc:290] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:50.466315 16528 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975) with higher term. Message: Invalid argument: T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 for earlier term 18. Current term is 19.
W20260709 13:04:50.475695  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:50.492936  9330 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:50.494916 16352 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:50.502146  9324 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:50.506295 16283 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:04:50.520092  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.529657  8654 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:50.531486 16227 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:04:50.574373  8474 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 309) took 28420 ms (client timeout 15000 ms). Trace:
W20260709 13:04:50.574851  8474 rpcz_store.cc:269] 0709 13:04:22.179405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.868270 (+1688865us) service_pool.cc:224] Handling call
0709 13:04:50.574275 (+26706005us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:50.336501 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:50.337034 (+   533us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:50.337067 (+    33us) write_op.cc:435] Acquired schema lock
0709 13:04:50.337084 (+    17us) tablet.cc:662] Decoding operations
0709 13:04:50.360751 (+ 23667us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:50.360843 (+    92us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:50.360871 (+    28us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:04:50.362033 (+  1162us) tablet.cc:701] Row locks acquired
0709 13:04:50.362052 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:04:50.362187 (+   135us) write_op.cc:270] Start()
0709 13:04:50.362296 (+   109us) write_op.cc:276] Timestamp: P: 1783602290362140 usec, L: 0
0709 13:04:50.362317 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:04:50.377667 (+ 15350us) log.cc:844] Serialized 6251 byte log entry
0709 13:04:50.436960 (+ 59293us) op_driver.cc:464] REPLICATION: finished
0709 13:04:50.496005 (+ 59045us) write_op.cc:301] APPLY: starting
0709 13:04:50.496092 (+    87us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:50.507093 (+ 11001us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:50.507189 (+    96us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:50.533980 (+ 26791us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:50.534399 (+   419us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 13:04:50.534467 (+    68us) write_op.cc:312] APPLY: finished
0709 13:04:50.534971 (+   504us) log.cc:844] Serialized 148 byte log entry
0709 13:04:50.545005 (+ 10034us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:50.545198 (+   193us) write_op.cc:454] Released schema lock
0709 13:04:50.545262 (+    64us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:50.545766 (+   504us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:50.545804 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":58712,"apply.run_cpu_time_us":21169,"apply.run_wall_time_us":49753,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70975,"num_ops":16,"prepare.queue_time_us":80424,"prepare.run_cpu_time_us":5124,"prepare.run_wall_time_us":49381,"replication_time_us":74432,"thread_start_us":77799,"threads_started":2}]]}
W20260709 13:04:50.629611  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.623001  8477 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:50.638002  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:50.678814  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:50.683872  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.745604  8473 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 13 to 18, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 18 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:50.748287  8473 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 11 to 15, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.756052  8473 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 13 to 16, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 16 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:50.750223  8477 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 308) took 56719 ms (client timeout 15000 ms). Trace:
W20260709 13:04:50.757517  8477 rpcz_store.cc:269] 0709 13:03:54.063863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.064595 (+   732us) service_pool.cc:224] Handling call
0709 13:04:50.750132 (+56685537us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:50.657753 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:50.658340 (+   587us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:50.658381 (+    41us) write_op.cc:435] Acquired schema lock
0709 13:04:50.658400 (+    19us) tablet.cc:662] Decoding operations
0709 13:04:50.658872 (+   472us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:50.658948 (+    76us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:50.658981 (+    33us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:04:50.659186 (+   205us) tablet.cc:701] Row locks acquired
0709 13:04:50.659206 (+    20us) write_op.cc:260] PREPARE: finished
0709 13:04:50.659364 (+   158us) write_op.cc:270] Start()
0709 13:04:50.659494 (+   130us) write_op.cc:276] Timestamp: P: 1783602290659308 usec, L: 0
0709 13:04:50.659519 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:04:50.660247 (+   728us) log.cc:844] Serialized 535 byte log entry
0709 13:04:50.662148 (+  1901us) op_driver.cc:464] REPLICATION: finished
0709 13:04:50.662658 (+   510us) write_op.cc:301] APPLY: starting
0709 13:04:50.662742 (+    84us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:50.663566 (+   824us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:50.663669 (+   103us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:50.664347 (+   678us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:50.664518 (+   171us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=2,mrs_lookups=0
0709 13:04:50.664567 (+    49us) write_op.cc:312] APPLY: finished
0709 13:04:50.664761 (+   194us) log.cc:844] Serialized 25 byte log entry
0709 13:04:50.664933 (+   172us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:50.665038 (+   105us) write_op.cc:454] Released schema lock
0709 13:04:50.665083 (+    45us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:50.665473 (+   390us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:50.665512 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1624473,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":278,"apply.run_cpu_time_us":2761,"apply.run_wall_time_us":2761,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4178,"num_ops":1,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":2781,"prepare.run_wall_time_us":2782,"replication_time_us":2490}]]}
W20260709 13:04:50.763036  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.769382  8473 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 8 to 16, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 16 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.772289  8473 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 9 to 18, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 18 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:50.787760  8473 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.792407  8473 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 15 to 17, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 17 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:50.804455  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:50.839833  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:50.889125  8477 catalog_manager.cc:5697] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 19, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 19 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.891490  8477 catalog_manager.cc:5697] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 21 to 23, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 23 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.893328  8477 catalog_manager.cc:5697] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 22 to 24, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 24 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.907722  8477 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.914958  8477 catalog_manager.cc:5697] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.916939  8477 catalog_manager.cc:5697] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 10 to 12, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.918821  8477 catalog_manager.cc:5697] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 20 to 23. New cstate: current_term: 23 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.920573  8477 catalog_manager.cc:5697] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.922086  8477 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 20, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 20 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:50.923202  8473 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 311) took 26594 ms (client timeout 15000 ms). Trace:
W20260709 13:04:50.925168  8473 rpcz_store.cc:269] 0709 13:04:24.320422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.729523 (+10409101us) service_pool.cc:224] Handling call
0709 13:04:50.914453 (+16184930us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:50.799102 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:50.799712 (+   610us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:50.799747 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:04:50.799767 (+    20us) tablet.cc:662] Decoding operations
0709 13:04:50.800664 (+   897us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:50.800733 (+    69us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:50.800768 (+    35us) tablet.cc:685] Acquiring locks for 7 operations
0709 13:04:50.801480 (+   712us) tablet.cc:701] Row locks acquired
0709 13:04:50.801496 (+    16us) write_op.cc:260] PREPARE: finished
0709 13:04:50.801643 (+   147us) write_op.cc:270] Start()
0709 13:04:50.801759 (+   116us) write_op.cc:276] Timestamp: P: 1783602290801592 usec, L: 0
0709 13:04:50.801776 (+    17us) op_driver.cc:348] REPLICATION: starting
0709 13:04:50.802474 (+   698us) log.cc:844] Serialized 2813 byte log entry
0709 13:04:50.809742 (+  7268us) op_driver.cc:464] REPLICATION: finished
0709 13:04:50.812568 (+  2826us) write_op.cc:301] APPLY: starting
0709 13:04:50.812666 (+    98us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:50.817273 (+  4607us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:50.817815 (+   542us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:50.822947 (+  5132us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:50.823197 (+   250us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=14,mrs_lookups=0
0709 13:04:50.823256 (+    59us) write_op.cc:312] APPLY: finished
0709 13:04:50.823599 (+   343us) log.cc:844] Serialized 73 byte log entry
0709 13:04:50.824579 (+   980us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:50.824718 (+   139us) write_op.cc:454] Released schema lock
0709 13:04:50.824770 (+    52us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:50.857658 (+ 32888us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:50.857701 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10360,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":2564,"apply.run_cpu_time_us":11448,"apply.run_wall_time_us":45037,"cfile_cache_hit":28,"cfile_cache_hit_bytes":29246,"num_ops":7,"prepare.queue_time_us":150,"prepare.run_cpu_time_us":3759,"prepare.run_wall_time_us":18600,"replication_time_us":7765}]]}
I20260709 13:04:50.924202  8477 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.931974  8477 catalog_manager.cc:5697] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 18, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.934516  8477 catalog_manager.cc:5697] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 21 to 22, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 22 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.936566  8477 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 16 to 21. New cstate: current_term: 21 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.938248  8477 catalog_manager.cc:5697] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 16, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 16 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.940171  8477 catalog_manager.cc:5697] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 17, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 17 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.941890  8477 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 18 to 19, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 19 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.946369  8477 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 13 to 17, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.948225  8477 catalog_manager.cc:5697] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.949864  8477 catalog_manager.cc:5697] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 21, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 21 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:50.951779  8477 catalog_manager.cc:5697] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.953780  8477 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 18, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 18 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.955520  8477 catalog_manager.cc:5697] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 18 to 21, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 21 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.957479  8477 catalog_manager.cc:5697] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 14 to 21. New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.959048  8477 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 17, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.960681  8477 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:50.962247  8477 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 12 to 15, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:51.049263 16364 raft_consensus.cc:493] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:51.049917 16364 raft_consensus.cc:515] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.052352 16364 leader_election.cc:290] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:51.054538  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.062433  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ffc6f04aee94f37b24daec463e6eb45" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.067186  8549 leader_election.cc:304] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.068430 16481 raft_consensus.cc:2749] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:51.114809  8477 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47036 (request call id 209) took 1285 ms. Trace:
I20260709 13:04:51.118881 16481 raft_consensus.cc:493] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.119529 16481 raft_consensus.cc:515] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:51.124982 16481 leader_election.cc:290] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:51.127506  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:51.115335  8477 rpcz_store.cc:276] 0709 13:04:49.825497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.762057 (+936560us) service_pool.cc:224] Handling call
0709 13:04:51.111158 (+349101us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:50.992768 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:50.996123 (+  3355us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:50.996164 (+    41us) write_op.cc:435] Acquired schema lock
0709 13:04:50.996189 (+    25us) tablet.cc:662] Decoding operations
0709 13:04:50.998416 (+  2227us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:50.998483 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:50.998517 (+    34us) tablet.cc:685] Acquiring locks for 26 operations
0709 13:04:51.000954 (+  2437us) tablet.cc:701] Row locks acquired
0709 13:04:51.000977 (+    23us) write_op.cc:260] PREPARE: finished
0709 13:04:51.001163 (+   186us) write_op.cc:270] Start()
0709 13:04:51.001293 (+   130us) write_op.cc:276] Timestamp: P: 1783602291001103 usec, L: 0
0709 13:04:51.001316 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:04:51.004878 (+  3562us) log.cc:844] Serialized 10013 byte log entry
0709 13:04:51.012406 (+  7528us) op_driver.cc:464] REPLICATION: finished
0709 13:04:51.013729 (+  1323us) write_op.cc:301] APPLY: starting
0709 13:04:51.013850 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:51.032521 (+ 18671us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:51.032644 (+   123us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:51.055427 (+ 22783us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:51.055827 (+   400us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=52,mrs_lookups=0
0709 13:04:51.055889 (+    62us) write_op.cc:312] APPLY: finished
0709 13:04:51.056428 (+   539us) log.cc:844] Serialized 228 byte log entry
0709 13:04:51.057307 (+   879us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:51.057573 (+   266us) write_op.cc:454] Released schema lock
0709 13:04:51.057647 (+    74us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:51.087339 (+ 29692us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:51.087381 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26034,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":1050,"apply.run_cpu_time_us":35873,"apply.run_wall_time_us":73536,"cfile_cache_hit":104,"cfile_cache_hit_bytes":108628,"num_ops":26,"prepare.queue_time_us":8000,"prepare.run_cpu_time_us":7253,"prepare.run_wall_time_us":31755,"replication_time_us":10904}]]}
I20260709 13:04:51.129041  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ad2aa6046e746348b02a2571b7580f0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.131353  8550 leader_election.cc:304] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:51.133237 16449 raft_consensus.cc:2749] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:51.122112  8478 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:51.169234 16481 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.169878 16481 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.182775 16481 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:51.183792  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.186666  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:51.202528  8622 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:51.205268  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.213615 16364 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:51.225849  8477 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 18 to 21, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 21 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:51.227401  8477 catalog_manager.cc:5697] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 16 to 19, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 19 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:51.228952  8477 catalog_manager.cc:5697] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 16 to 23. New cstate: current_term: 23 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:51.230635  8477 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:51.232018  8477 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 18 to 23, leader changed from 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 23 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:51.287408 16364 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.288017 16364 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.292919  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.305685 16364 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:51.306777  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaeef2720084b6aac897de8d33f6921" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.308553  8550 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.309808 16364 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:51.334357  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:51.348182  8478 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 310) took 49491 ms (client timeout 15000 ms). Trace:
W20260709 13:04:51.361534  8478 rpcz_store.cc:269] 0709 13:04:01.857885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.858950 (+  1065us) service_pool.cc:224] Handling call
0709 13:04:51.220084 (+49361134us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:51.220116 (+    32us) write_op.cc:454] Released schema lock
0709 13:04:51.348096 (+127980us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:51.204228 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:51.204813 (+   585us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:51.204857 (+    44us) write_op.cc:435] Acquired schema lock
0709 13:04:51.204872 (+    15us) tablet.cc:662] Decoding operations
0709 13:04:51.207217 (+  2345us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:51.207281 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:51.207318 (+    37us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:04:51.207537 (+   219us) tablet.cc:701] Row locks acquired
0709 13:04:51.207550 (+    13us) write_op.cc:260] PREPARE: finished
0709 13:04:51.207694 (+   144us) write_op.cc:270] Start()
0709 13:04:51.207807 (+   113us) write_op.cc:276] Timestamp: P: 1783602291207649 usec, L: 0
0709 13:04:51.207832 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:04:51.208478 (+   646us) log.cc:844] Serialized 535 byte log entry
0709 13:04:51.210605 (+  2127us) op_driver.cc:464] REPLICATION: finished
0709 13:04:51.211030 (+   425us) write_op.cc:301] APPLY: starting
0709 13:04:51.211122 (+    92us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:51.211995 (+   873us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:51.212107 (+   112us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:51.212868 (+   761us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:51.213033 (+   165us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=2,mrs_lookups=0
0709 13:04:51.213094 (+    61us) write_op.cc:312] APPLY: finished
0709 13:04:51.213290 (+   196us) log.cc:844] Serialized 25 byte log entry
0709 13:04:51.216988 (+  3698us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:51.217091 (+   103us) write_op.cc:454] Released schema lock
0709 13:04:51.217136 (+    45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":104,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":224,"apply.run_cpu_time_us":2918,"apply.run_wall_time_us":6411,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4178,"num_ops":1,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":2620,"prepare.run_wall_time_us":4534,"replication_time_us":2651}]]}
W20260709 13:04:51.394917  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:51.398729  7481 thread.cc:527] Waited for 6000ms trying to join with heartbeat (tid 8609)
I20260709 13:04:51.437209 16364 raft_consensus.cc:493] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.437909 16364 raft_consensus.cc:515] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:51.442960  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:51.446142  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b06c2e2a7c644478ef688b47c11a07a" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.447506 16481 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:51.447705  8549 leader_election.cc:304] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.448108 16481 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:51.448457 16449 raft_consensus.cc:2749] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:51.449474 16364 leader_election.cc:290] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:51.458859  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.460420  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03f3a12327224113b1c5bf090e280a53" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:51.462527  8548 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:51.476375 16364 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:51.450412 16481 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:04:51.480293  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:51.483811  8479 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 305) took 78464 ms (client timeout 15000 ms). Trace:
W20260709 13:04:51.487283  8479 rpcz_store.cc:269] 0709 13:03:33.018978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.020021 (+  1043us) service_pool.cc:224] Handling call
0709 13:04:51.483612 (+78463591us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99,"reported_tablets":47,"spinlock_wait_cycles":1920}
I20260709 13:04:51.506592 16449 raft_consensus.cc:493] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.507170 16449 raft_consensus.cc:515] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:51.512701  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:51.514600 16449 leader_election.cc:290] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:51.516103  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5559db905acf4ba0b48e9e3e87f49c1c" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.518458  8550 leader_election.cc:304] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:51.519587 16449 raft_consensus.cc:2749] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:51.530845 16449 raft_consensus.cc:493] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:04:51.531418 16449 raft_consensus.cc:515] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.533595 16449 leader_election.cc:290] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:51.535158  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.544451  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01467ec5f29c4a728b24e55d1f5d2e94" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.549341  8549 leader_election.cc:304] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.559036 16449 raft_consensus.cc:2749] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:51.561750 16449 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.562309 16449 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.564531 16449 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:51.566110 16449 raft_consensus.cc:493] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.566623 16449 raft_consensus.cc:515] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.568827 16449 leader_election.cc:290] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:51.581471  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.582861  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.584331  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "121ce91dbce74f019e528db7b7062a86" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.586673  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06f98a166dd945b39554b2f278c933ef" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.588490  8550 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.589798 16481 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:51.591174  8550 leader_election.cc:304] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.592269 16449 raft_consensus.cc:2749] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:51.615263  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:51.643651  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:51.653797 16449 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.654389 16449 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:51.656507 16449 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:51.658900  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.660677  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9b3791a4a534a598bc36d62992d27be" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
W20260709 13:04:51.661842  8480 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 306) took 108341 ms (client timeout 15000 ms). Trace:
I20260709 13:04:51.662449  8548 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:51.663743 16449 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:51.662307  8480 rpcz_store.cc:269] 0709 13:03:03.320621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.369489 (+ 48868us) service_pool.cc:224] Handling call
0709 13:04:51.661620 (+108292131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":951852,"reported_tablets":46}
W20260709 13:04:51.673843  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet 7eaeef2720084b6aac897de8d33f6921: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 19 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:51.682996  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet b063565a0d8244cabf980570d2fc8a38: Illegal state: Replica 1dc41db601fe4b91aa6d461b504da0e7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:51.758867  8481 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 308) took 101806 ms (client timeout 15000 ms). Trace:
W20260709 13:04:51.776624  8481 rpcz_store.cc:269] 0709 13:03:09.952918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:09.953604 (+   686us) service_pool.cc:224] Handling call
0709 13:04:51.758659 (+101805055us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":233666,"reported_tablets":47}
I20260709 13:04:51.789688  8475 catalog_manager.cc:5697] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 15 to 20. New cstate: current_term: 20 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.791376  8475 catalog_manager.cc:5697] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 18 to 20, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 20 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.797048  8475 catalog_manager.cc:5697] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:51.792841  8482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 307) took 134256 ms (client timeout 15000 ms). Trace:
I20260709 13:04:51.798408 16364 raft_consensus.cc:493] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.800836 16364 raft_consensus.cc:515] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.802978 16364 leader_election.cc:290] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:51.803896  8482 rpcz_store.cc:269] 0709 13:02:37.548324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.557640 (+  9316us) service_pool.cc:224] Handling call
0709 13:02:37.856603 (+298963us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b540012a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b80090 00007f1409c1ef3a 00007f140f9d51cc 00007f140fad337e 00007f140df942a4 00007f140df94232 00007f140df94161 00007f140df940dc 00007f140df93392 00007f1407f5148c 00007f1407f508c6
0709 13:04:51.792647 (+133936044us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1288770,"reported_tablets":37,"spinlock_wait_cycles":24059264}
I20260709 13:04:51.805531  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.807296  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9413ba3105764883ae0d7597a86ad605" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.808154  8475 catalog_manager.cc:5697] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 13 to 18. New cstate: current_term: 18 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.809026  8549 leader_election.cc:304] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.812318 16364 raft_consensus.cc:2749] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:51.816622  8475 catalog_manager.cc:5697] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 18 to 19, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 19 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:51.826603  8475 catalog_manager.cc:5697] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 16 to 20, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 20 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.829391  8475 catalog_manager.cc:5697] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 15 to 20. New cstate: current_term: 20 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:51.839469  8475 catalog_manager.cc:5697] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 14 to 17, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:51.841534 16481 raft_consensus.cc:493] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.853758 16481 raft_consensus.cc:515] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:51.855940 16481 leader_election.cc:290] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:51.845242  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet 652ad3041b404044aaf40f77c1be3a32: Illegal state: Replica d0a2b9c938d94a4fa258e068ea054d84 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.857784  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:51.863281  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4572d49b22942189c377be6ba2293d6" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.864580  8549 leader_election.cc:304] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.865561 16481 raft_consensus.cc:2749] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:51.865576  8475 catalog_manager.cc:5697] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 18, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 18 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.871309  8475 catalog_manager.cc:5697] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 22. New cstate: current_term: 22 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260709 13:04:51.873144  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet 6b93c5bfc29f4210b1fa8340091624eb: Illegal state: Replica d0a2b9c938d94a4fa258e068ea054d84 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.878926  8475 catalog_manager.cc:5697] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 16 to 23. New cstate: current_term: 23 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:51.880507  8475 catalog_manager.cc:5697] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 21. New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.881790  8475 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 15 to 16, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 16 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.883255  8475 catalog_manager.cc:5697] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 18, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 18 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.884776  8475 catalog_manager.cc:5697] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 21, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.886353  8475 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 19, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 19 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:51.918505 16481 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.919162 16481 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:51.922487  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:51.926226  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b387b386886048c8b66e5f499f02aab2" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:51.928004  8549 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:51.928871 16481 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:51.928865 16364 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:52.019656 16481 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:52.020787  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.020231 16481 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:52.022912  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.025115  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:52.025681  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1f77a3af1d2472a857b0f7a57069d60" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:52.027360 16481 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:52.028311 16481 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:52.029228 16481 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:52.030884  8549 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.031780 16364 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:52.033224  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:52.033860  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde36bf9fcc5409e8a00b747fbfbfb06" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:52.036787 16481 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:52.038709  8548 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.039662 16481 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:52.040925  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:52.111855  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.214442  8476 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 15 to 22, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 22 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.218528  8476 catalog_manager.cc:5697] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 16 to 19, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 19 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.220678  8476 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.221906  8475 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38104 (request call id 312) took 7103 ms. Trace:
W20260709 13:04:52.234884  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:52.238459  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.225627  8475 rpcz_store.cc:276] 0709 13:04:45.134167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:47.296581 (+2162414us) service_pool.cc:224] Handling call
0709 13:04:48.134731 (+838150us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b540012a554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b80090 00007f1409c1ef3a 00007f140f9d51cc 00007f140fad337e 00007f140df942a4 00007f140df94232 00007f140df94161 00007f140df940dc 00007f140df93392 00007f1407f5148c 00007f1407f508c6
0709 13:04:52.221832 (+4087101us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:51.930491 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:51.931088 (+   597us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:51.931129 (+    41us) write_op.cc:435] Acquired schema lock
0709 13:04:51.931155 (+    26us) tablet.cc:662] Decoding operations
0709 13:04:51.932689 (+  1534us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:51.932792 (+   103us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:51.932825 (+    33us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:04:51.934169 (+  1344us) tablet.cc:701] Row locks acquired
0709 13:04:51.934190 (+    21us) write_op.cc:260] PREPARE: finished
0709 13:04:51.934353 (+   163us) write_op.cc:270] Start()
0709 13:04:51.934478 (+   125us) write_op.cc:276] Timestamp: P: 1783602291934293 usec, L: 0
0709 13:04:51.934500 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:04:51.935266 (+   766us) log.cc:844] Serialized 6266 byte log entry
0709 13:04:51.978911 (+ 43645us) op_driver.cc:464] REPLICATION: finished
0709 13:04:51.986667 (+  7756us) write_op.cc:301] APPLY: starting
0709 13:04:51.986764 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:52.075882 (+ 89118us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:52.076027 (+   145us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:52.111886 (+ 35859us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:52.112197 (+   311us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 13:04:52.112244 (+    47us) write_op.cc:312] APPLY: finished
0709 13:04:52.112642 (+   398us) log.cc:844] Serialized 148 byte log entry
0709 13:04:52.112812 (+   170us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:52.112979 (+   167us) write_op.cc:454] Released schema lock
0709 13:04:52.113030 (+    51us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:52.113441 (+   411us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:52.113472 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":48,"spinlock_wait_cycles":4791040,"child_traces":[["op",{"apply.queue_time_us":7455,"apply.run_cpu_time_us":22402,"apply.run_wall_time_us":126748,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70975,"num_ops":16,"prepare.queue_time_us":29000,"prepare.run_cpu_time_us":5160,"prepare.run_wall_time_us":12347,"replication_time_us":44233,"spinlock_wait_cycles":54528,"thread_start_us":35849,"threads_started":2}]]}
I20260709 13:04:52.273561  8476 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 12 to 17, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 17 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:52.286309 16481 raft_consensus.cc:493] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.287025 16481 raft_consensus.cc:515] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:52.293058  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:52.293239  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b894d0fc374f33a626d5fd9d8ca4e7" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:52.295401  8548 leader_election.cc:304] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:52.296444 16364 raft_consensus.cc:2749] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:52.297426 16364 raft_consensus.cc:493] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.297953 16364 raft_consensus.cc:515] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:52.300504 16364 leader_election.cc:290] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:52.302114  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:52.309260  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90d5e31850d64b9eb8c754954c889846" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:52.310458  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:52.315079 16481 leader_election.cc:290] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:52.318339  8549 leader_election.cc:304] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.324566 16481 raft_consensus.cc:2749] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:52.327435  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.281661  8476 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 16 to 18, leader changed from 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 18 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:52.399386  7481 thread.cc:527] Waited for 7000ms trying to join with heartbeat (tid 8609)
W20260709 13:04:52.410887  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.429944  8475 catalog_manager.cc:5697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 20, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to a0d703ee18464475a290425976246555 (127.7.78.65). New cstate: current_term: 20 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:52.431805  8475 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:52.435727  8475 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:52.445700  8476 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38106 (request call id 308) took 7074 ms. Trace:
W20260709 13:04:52.445700  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.437147  8475 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "a0d703ee18464475a290425976246555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
W20260709 13:04:52.455564  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.446151  8476 rpcz_store.cc:276] 0709 13:04:45.371277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.068806 (+4697529us) service_pool.cc:224] Handling call
0709 13:04:52.445614 (+2376808us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:52.349904 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:52.350416 (+   512us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:52.350453 (+    37us) write_op.cc:435] Acquired schema lock
0709 13:04:52.350465 (+    12us) tablet.cc:662] Decoding operations
0709 13:04:52.351112 (+   647us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:52.351162 (+    50us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:52.351195 (+    33us) tablet.cc:685] Acquiring locks for 5 operations
0709 13:04:52.351703 (+   508us) tablet.cc:701] Row locks acquired
0709 13:04:52.351722 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:04:52.351860 (+   138us) write_op.cc:270] Start()
0709 13:04:52.351965 (+   105us) write_op.cc:276] Timestamp: P: 1783602292351815 usec, L: 0
0709 13:04:52.351986 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 13:04:52.352585 (+   599us) log.cc:844] Serialized 2064 byte log entry
0709 13:04:52.354660 (+  2075us) op_driver.cc:464] REPLICATION: finished
0709 13:04:52.358409 (+  3749us) write_op.cc:301] APPLY: starting
0709 13:04:52.358524 (+   115us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:52.361832 (+  3308us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:52.361957 (+   125us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:52.365831 (+  3874us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:52.366095 (+   264us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=10,mrs_lookups=0
0709 13:04:52.366161 (+    66us) write_op.cc:312] APPLY: finished
0709 13:04:52.366453 (+   292us) log.cc:844] Serialized 57 byte log entry
0709 13:04:52.367326 (+   873us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:52.367452 (+   126us) write_op.cc:454] Released schema lock
0709 13:04:52.367514 (+    62us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:52.367867 (+   353us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:52.367902 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":40019,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":3531,"apply.run_cpu_time_us":8744,"apply.run_wall_time_us":9425,"cfile_cache_hit":20,"cfile_cache_hit_bytes":20890,"num_ops":5,"prepare.queue_time_us":2015,"prepare.run_cpu_time_us":2939,"prepare.run_wall_time_us":2938,"replication_time_us":2570}]]}
W20260709 13:04:52.490908  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.548233  8474 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38096 (request call id 310) took 6024 ms. Trace:
I20260709 13:04:52.549281  8474 rpcz_store.cc:276] 0709 13:04:46.523689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.578431 (+4054742us) service_pool.cc:224] Handling call
0709 13:04:52.548132 (+1969701us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":47}
I20260709 13:04:52.565771  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:04:52.569892  7481 tablet_replica.cc:333] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.571048  7481 raft_consensus.cc:2243] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Raft consensus shutting down.
W20260709 13:04:52.573330  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:52.575083  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.579422  7481 raft_consensus.cc:2272] T 4d18196b7e714cb6a79ec64ea3d65803 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.583395  7481 tablet_replica.cc:333] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.588954  7481 raft_consensus.cc:2243] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.595023  7481 raft_consensus.cc:2272] T 90d5e31850d64b9eb8c754954c889846 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.607614  7481 tablet_replica.cc:333] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.608796  7481 raft_consensus.cc:2243] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus shutting down.
W20260709 13:04:52.620231  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.629247  7481 pending_rounds.cc:70] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:52.634121  7481 pending_rounds.cc:77] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 18 index: 9 } timestamp: 7305634894217719808 op_type: NO_OP noop_request { }
I20260709 13:04:52.634738  7481 raft_consensus.cc:2889] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:52.635227  7481 raft_consensus.cc:2272] T 96cbbc16390646a9875b38f1b174f981 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.652769  7481 tablet_replica.cc:333] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.653844  7481 raft_consensus.cc:2243] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.654405  7481 raft_consensus.cc:2272] T b8b894d0fc374f33a626d5fd9d8ca4e7 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.675055  7481 tablet_replica.cc:333] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.675112 16364 raft_consensus.cc:493] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.676120  7481 raft_consensus.cc:2243] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.675807 16364 raft_consensus.cc:515] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:52.677630  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:52.678144 16364 leader_election.cc:290] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:52.679203  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:52.679288  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a6a2506bc27408a9b9113a0fa1f069f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:52.682150  8549 leader_election.cc:304] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.682966 16364 raft_consensus.cc:2749] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:52.686259  7481 pending_rounds.cc:70] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:52.686537  7481 pending_rounds.cc:77] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 18 index: 10 } timestamp: 7305634950987206656 op_type: NO_OP noop_request { }
I20260709 13:04:52.686905  7481 raft_consensus.cc:2889] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:52.687243  7481 raft_consensus.cc:2272] T 9413ba3105764883ae0d7597a86ad605 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:52.691884  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.694703  7481 tablet_replica.cc:333] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.695677  7481 raft_consensus.cc:2243] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:04:52.712350  7481 raft_consensus.cc:2272] T 9a389d19cf2c47d6b5873139d8cfad40 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.723290  7481 tablet_replica.cc:333] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.735692  7481 raft_consensus.cc:2243] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus shutting down.
W20260709 13:04:52.738579  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.736400  7481 raft_consensus.cc:2272] T 7eaeef2720084b6aac897de8d33f6921 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.742877  7481 tablet_replica.cc:333] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.743897  7481 raft_consensus.cc:2243] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:04:52.744907  7481 raft_consensus.cc:2272] T d851286f2cf248549fc81f8296784c08 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.748003 16364 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:04:52.748586 16364 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:52.750881 16364 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:52.753507  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:52.756362  8653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a346f8961f774e87a3c75fa7e2053cb0" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:52.760689  8549 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.766671 16364 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:52.778221 16364 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.778828 16364 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:52.783463  9330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:52.787947 16364 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:52.788875  7481 tablet_replica.cc:333] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.790102 16364 raft_consensus.cc:493] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.790902 16364 raft_consensus.cc:515] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:52.794334  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:52.796378 16364 leader_election.cc:290] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:52.798580  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2adcf8a687b54060b644e8c691d8772f" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:52.799142  7481 raft_consensus.cc:2243] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.799940  7481 raft_consensus.cc:2272] T 9f9dd5b88e0647f284658784381e1893 P a0d703ee18464475a290425976246555 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.800185  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "652ad3041b404044aaf40f77c1be3a32" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:52.802934  7481 tablet_replica.cc:333] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.804003  7481 raft_consensus.cc:2243] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.804682  7481 raft_consensus.cc:2272] T 081470ef13974fe497f88859ada43ffa P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.820905  7481 tablet_replica.cc:333] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.821882  7481 raft_consensus.cc:2243] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:04:52.822985  7481 raft_consensus.cc:2272] T 96614bb40dc54bb2bdcff8f9d5476bb1 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.833921  8549 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.834779 16364 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:52.835202  8549 leader_election.cc:304] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.835959 16364 raft_consensus.cc:2749] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:52.853956  7481 tablet_replica.cc:333] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.855037  7481 raft_consensus.cc:2243] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.855764  7481 raft_consensus.cc:2272] T b4572d49b22942189c377be6ba2293d6 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.862133  7481 tablet_replica.cc:333] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.863049  7481 raft_consensus.cc:2243] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.863792  7481 raft_consensus.cc:2272] T 294596405ca140769bf30fa7ee1e4820 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.887012  7481 tablet_replica.cc:333] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.887948  7481 raft_consensus.cc:2243] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.888540  7481 raft_consensus.cc:2272] T 7778779e8695473fa6249308e7a4fa1e P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.891678  7481 tablet_replica.cc:333] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.892684  7481 raft_consensus.cc:2243] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.911917 16481 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.912484 16481 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:52.915341 16481 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:52.915885  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:52.893296  7481 raft_consensus.cc:2272] T 0337bfcbeb3b4e15b96669c8d3ca4692 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.917572  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94c0917e28b9498a823f74b5f664a815" candidate_uuid: "a0d703ee18464475a290425976246555" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:52.919308  8548 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0d703ee18464475a290425976246555; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:52.920331  7481 tablet_replica.cc:333] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.920405 16481 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:52.921442  7481 raft_consensus.cc:2243] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.922367  7481 raft_consensus.cc:2272] T f4f350f118fc449fb61357528a6b14f7 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:52.940073  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:52.945695  7481 tablet_replica.cc:333] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.946784  7481 raft_consensus.cc:2243] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.947537  7481 pending_rounds.cc:70] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:52.947808  7481 pending_rounds.cc:77] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 19 index: 14 } timestamp: 7305634894368636928 op_type: NO_OP noop_request { }
I20260709 13:04:52.948146  7481 raft_consensus.cc:2889] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:52.948503  7481 raft_consensus.cc:2272] T 06f98a166dd945b39554b2f278c933ef P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.951498  7481 tablet_replica.cc:333] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.952323  7481 raft_consensus.cc:2243] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:04:52.953423  7481 raft_consensus.cc:2272] T bab857d69b5f4bc480340a9324923bc4 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.956278  7481 tablet_replica.cc:333] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.957101  7481 raft_consensus.cc:2243] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.973757  7481 raft_consensus.cc:2272] T 3ad2aa6046e746348b02a2571b7580f0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.976863  7481 tablet_replica.cc:333] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.977743  7481 raft_consensus.cc:2243] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:04:52.978843  7481 raft_consensus.cc:2272] T 9e8a9411169f407cba1b44443edc9840 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:52.986605  7481 tablet_replica.cc:333] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:52.987640  7481 raft_consensus.cc:2243] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:52.988291  7481 pending_rounds.cc:70] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:52.988548  7481 pending_rounds.cc:77] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 18 index: 11 } timestamp: 7305634901913088000 op_type: NO_OP noop_request { }
I20260709 13:04:52.988925  7481 raft_consensus.cc:2889] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:52.989289  7481 raft_consensus.cc:2272] T f057517aa4a1496ab40134759e3d8b54 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.004477  7481 tablet_replica.cc:333] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.005425  7481 raft_consensus.cc:2243] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:04:53.006587  7481 raft_consensus.cc:2272] T 266236cfcd824409b757505321b36d81 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.019941  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.040915  7481 tablet_replica.cc:333] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555: stopping tablet replica
W20260709 13:04:53.043763  8696 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.050349  7481 raft_consensus.cc:2243] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.051106  7481 raft_consensus.cc:2272] T 689ad8326d634dee8402f1352a76f0e2 P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.054612  7481 tablet_replica.cc:333] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.055474  7481 raft_consensus.cc:2243] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.056218  7481 raft_consensus.cc:2272] T 3a6a2506bc27408a9b9113a0fa1f069f P a0d703ee18464475a290425976246555 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.059240  7481 tablet_replica.cc:333] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.060078  7481 raft_consensus.cc:2243] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.060680  7481 pending_rounds.cc:70] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:53.060915  7481 pending_rounds.cc:77] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 15 index: 11 } timestamp: 7305634908131864576 op_type: NO_OP noop_request { }
I20260709 13:04:53.084368  7481 raft_consensus.cc:2889] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:53.084821  7481 raft_consensus.cc:2272] T 01467ec5f29c4a728b24e55d1f5d2e94 P a0d703ee18464475a290425976246555 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.088133  7481 tablet_replica.cc:333] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.089162  7481 raft_consensus.cc:2243] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.089825  7481 raft_consensus.cc:2272] T 121ce91dbce74f019e528db7b7062a86 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.092803  7481 tablet_replica.cc:333] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555: stopping tablet replica
W20260709 13:04:53.096182  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.109961  7481 raft_consensus.cc:2243] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.110782  7481 raft_consensus.cc:2272] T 5559db905acf4ba0b48e9e3e87f49c1c P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.137790  7481 tablet_replica.cc:333] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.138646  7481 raft_consensus.cc:2243] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.139389  7481 raft_consensus.cc:2272] T a346f8961f774e87a3c75fa7e2053cb0 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.156850  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.158285  7481 tablet_replica.cc:333] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.159256  7481 raft_consensus.cc:2243] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.159822  7481 raft_consensus.cc:2272] T a7c87eef075e49ba98e563e6fcc45d83 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.162649  7481 tablet_replica.cc:333] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.163477  7481 raft_consensus.cc:2243] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.164017  7481 raft_consensus.cc:2272] T 2adcf8a687b54060b644e8c691d8772f P a0d703ee18464475a290425976246555 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.167526  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.178862  7481 tablet_replica.cc:333] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.178509  8731 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:04:53.179739  7481 raft_consensus.cc:2243] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.185818  7481 raft_consensus.cc:2272] T eeef0c4282c74b3692ab849adb5ce58e P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.189266  7481 tablet_replica.cc:333] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.190097  7481 raft_consensus.cc:2243] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.190774  7481 raft_consensus.cc:2272] T 94c0917e28b9498a823f74b5f664a815 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.206202  7481 tablet_replica.cc:333] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.207082  7481 raft_consensus.cc:2243] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.207579  7481 raft_consensus.cc:2272] T 04399390cf414886a3453981dc1f74aa P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.210381  7481 tablet_replica.cc:333] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555: stopping tablet replica
W20260709 13:04:53.210992  8697 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7778779e8695473fa6249308e7a4fa1e. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:53.211138  7481 raft_consensus.cc:2243] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:04:53.212201  7481 raft_consensus.cc:2272] T a6c36d18684a44799bfb89a6d20e4fd7 P a0d703ee18464475a290425976246555 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.213757  8657 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:04:53.215788 16227 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:53.230988  7481 tablet_replica.cc:333] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.231897  7481 raft_consensus.cc:2243] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.232613  7481 raft_consensus.cc:2272] T 4b06c2e2a7c644478ef688b47c11a07a P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.233852  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:53.235667  7481 tablet_replica.cc:333] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555: stopping tablet replica
W20260709 13:04:53.236256  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.236800  7481 raft_consensus.cc:2243] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.242681 16549 raft_consensus.cc:2955] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Committing config change with OpId 21.14: config changed from index -1 to 14, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:04:53.253715  7481 raft_consensus.cc:2272] T cde36bf9fcc5409e8a00b747fbfbfb06 P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.256767  7481 tablet_replica.cc:333] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.256587  8653 raft_consensus.cc:2955] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Committing config change with OpId 21.14: config changed from index -1 to 14, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:04:53.257616  7481 raft_consensus.cc:2243] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.270051  7481 pending_rounds.cc:70] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:53.270344  7481 pending_rounds.cc:77] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 20 index: 13 } timestamp: 7305634900969734144 op_type: NO_OP noop_request { }
I20260709 13:04:53.270685  7481 raft_consensus.cc:2889] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:53.271035  7481 raft_consensus.cc:2272] T 652ad3041b404044aaf40f77c1be3a32 P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.273864  7481 tablet_replica.cc:333] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.274667  7481 raft_consensus.cc:2243] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:04:53.275698  7481 raft_consensus.cc:2272] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P a0d703ee18464475a290425976246555 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.285633  8731 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:04:53.292398  8474 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 14, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:04:53.302578  7481 tablet_replica.cc:333] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.303478  7481 raft_consensus.cc:2243] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.304116  7481 raft_consensus.cc:2272] T 03f3a12327224113b1c5bf090e280a53 P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.305435  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:53.306121  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.307085  7481 tablet_replica.cc:333] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.307886  7481 raft_consensus.cc:2243] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.308580  7481 pending_rounds.cc:70] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:53.308861  7481 pending_rounds.cc:77] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 20 index: 14 } timestamp: 7305634900629450752 op_type: NO_OP noop_request { }
I20260709 13:04:53.309233  7481 raft_consensus.cc:2889] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:53.325685  7481 raft_consensus.cc:2272] T eed0f0ed87f74c1fb8320d38bceeac72 P a0d703ee18464475a290425976246555 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.329895  7481 tablet_replica.cc:333] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.331773  7481 raft_consensus.cc:2243] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:04:53.332999  7481 raft_consensus.cc:2272] T b063565a0d8244cabf980570d2fc8a38 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.333802  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.335595  7481 tablet_replica.cc:333] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.336436  7481 raft_consensus.cc:2243] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.336962  7481 pending_rounds.cc:70] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Trying to abort 1 pending ops.
I20260709 13:04:53.337169  7481 pending_rounds.cc:77] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555: Aborting op as it isn't in flight: id { term: 16 index: 14 } timestamp: 7305634953534287872 op_type: NO_OP noop_request { }
I20260709 13:04:53.353612  7481 raft_consensus.cc:2889] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:53.353999  7481 raft_consensus.cc:2272] T b9b3791a4a534a598bc36d62992d27be P a0d703ee18464475a290425976246555 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.356693  7481 tablet_replica.cc:333] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.357436  7481 raft_consensus.cc:2243] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.358147  7481 raft_consensus.cc:2272] T 8ffc6f04aee94f37b24daec463e6eb45 P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.360909  7481 tablet_replica.cc:333] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.375883  8653 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 13:04:53.377739  7481 raft_consensus.cc:2243] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 16 LEADER]: Raft consensus shutting down.
I20260709 13:04:53.378968  7481 raft_consensus.cc:2272] T 6b93c5bfc29f4210b1fa8340091624eb P a0d703ee18464475a290425976246555 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.379055 16549 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 13:04:53.381618  8696 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:53.391245  7481 tablet_replica.cc:333] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555: stopping tablet replica
W20260709 13:04:53.391717  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.392275  7481 raft_consensus.cc:2243] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 20 LEADER]: Raft consensus shutting down.
W20260709 13:04:53.401172  8697 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133): Couldn't send request to peer 453e644bad5d410ab9be85b353e28dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 121ce91dbce74f019e528db7b7062a86. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:53.393373  7481 raft_consensus.cc:2272] T 98c31f9999e84270898dfb87c21cbc7c P a0d703ee18464475a290425976246555 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:53.406077  7481 tablet_replica.cc:333] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.406765  7481 raft_consensus.cc:2243] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.407485  7481 raft_consensus.cc:2272] T d1f77a3af1d2472a857b0f7a57069d60 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.418596 16553 log.cc:927] Time spent T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 13:04:53.423702 16549 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Committing config change with OpId 22.17: config changed from index -1 to 17, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:04:53.426178  8657 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Committing config change with OpId 22.17: config changed from index -1 to 17, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
I20260709 13:04:53.426244  7481 tablet_replica.cc:333] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555: stopping tablet replica
I20260709 13:04:53.466318  7481 raft_consensus.cc:2243] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:53.466876  7481 raft_consensus.cc:2272] T b387b386886048c8b66e5f499f02aab2 P a0d703ee18464475a290425976246555 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:53.519220  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet d851286f2cf248549fc81f8296784c08: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:53.522938  8530 auto_rebalancer.cc:629] Failed to schedule move for tablet b063565a0d8244cabf980570d2fc8a38: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.563565  8475 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index -1 to 17, NON_VOTER 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) added. New cstate: current_term: 22 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } } }
W20260709 13:04:53.727020  8696 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 345 similar messages]
W20260709 13:04:53.728825  8622 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.758477 16560 ts_tablet_manager.cc:933] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:53.768594 16554 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:53.769227 16554 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:53.772478 16554 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:53.774243  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:53.783454 16560 tablet_copy_client.cc:323] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:04:53.786091  8742 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 121ce91dbce74f019e528db7b7062a86 from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:04:53.793260  8742 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 121ce91dbce74f019e528db7b7062a86 from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-121ce91dbce74f019e528db7b7062a86
I20260709 13:04:53.815382  8742 tablet_copy_source_session.cc:215] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:04:53.841140 16560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121ce91dbce74f019e528db7b7062a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:04:53.841449 16562 ts_tablet_manager.cc:933] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: Initiating tablet copy from peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:53.854518 16549 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:53.855139 16549 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:53.858495 16549 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:53.860219  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
W20260709 13:04:53.873802  8696 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:53.874316  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:53.880012  9294 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:53.880553  9294 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:04:53.881527  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:53.882251 16554 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:53.889601 16549 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:53.898635  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:53.903759 16227 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:53.904382 16227 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:53.906924 16227 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:53.908330  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:53.909052  9323 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:04:53.910413  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:53.911351 16549 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:53.911791 16549 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:53.913415 16549 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:53.912495 16562 tablet_copy_client.cc:323] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.78.67:37975
I20260709 13:04:53.917999  8742 tablet_copy_service.cc:140] P 1dc41db601fe4b91aa6d461b504da0e7: Received BeginTabletCopySession request for tablet 7778779e8695473fa6249308e7a4fa1e from peer 453e644bad5d410ab9be85b353e28dc2 ({username='slave'} at 127.0.0.1:42702)
I20260709 13:04:53.918586  8742 tablet_copy_service.cc:161] P 1dc41db601fe4b91aa6d461b504da0e7: Beginning new tablet copy session on tablet 7778779e8695473fa6249308e7a4fa1e from peer 453e644bad5d410ab9be85b353e28dc2 at {username='slave'} at 127.0.0.1:42702: session id = 453e644bad5d410ab9be85b353e28dc2-7778779e8695473fa6249308e7a4fa1e
W20260709 13:04:53.924660  8696 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:53.927492  8742 tablet_copy_source_session.cc:215] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:04:53.944686 16562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7778779e8695473fa6249308e7a4fa1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:04:53.944679 16560 tablet_copy_client.cc:806] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:04:53.948307 16560 tablet_copy_client.cc:670] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:04:53.954393 16560 tablet_copy_client.cc:538] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:04:53.964718 16549 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:53.967981  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a389d19cf2c47d6b5873139d8cfad40" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:53.968611  9323 raft_consensus.cc:3060] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:53.974936 16560 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
W20260709 13:04:53.975162  8696 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:53.977514  9323 raft_consensus.cc:2468] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 22.
I20260709 13:04:53.979048  8697 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:53.981652 16352 raft_consensus.cc:2804] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:53.982146 16352 raft_consensus.cc:697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:53.983022 16352 consensus_queue.cc:237] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:54.002153  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:54.007889 16549 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:54.019256  8475 catalog_manager.cc:5697] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 21 to 22, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 22 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:54.022395 16562 tablet_copy_client.cc:806] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 0 data blocks...
I20260709 13:04:54.023103 16562 tablet_copy_client.cc:670] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Starting download of 1 WAL segments...
I20260709 13:04:54.035164 16562 tablet_copy_client.cc:538] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:04:54.060801 16352 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.061486 16352 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.064222 16352 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:54.069362  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:54.076890 16548 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.077837 16548 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:54.080740  8696 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.088155  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:04:54.089172  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:54.083627  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:54.091130 16227 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:54.081192 16548 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.100654 16558 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.101311 16558 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:54.101611  9294 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.108124  9294 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:04:54.110129 16548 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:54.112609  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:54.113359  8728 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
W20260709 13:04:54.113953  8622 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.114884  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:54.116377 16283 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:54.116775 16283 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.117134 16283 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:54.123122 16227 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.123677 16227 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.127626  8657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:54.129706 16558 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.134935 16283 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.137125 16227 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:54.127745 16562 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: Bootstrap starting.
W20260709 13:04:54.174690  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.175235  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:54.179351 16283 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.180244 16549 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:54.266857 16548 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.268929 16548 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.322304 16548 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:54.356236  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 50100.9us
W20260709 13:04:54.357947  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:54.358639  9294 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.360829  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:54.363102  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:54.366763 16554 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:54.390133 16572 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.390894 16572 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.413681 16572 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:54.418991  8622 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.423439  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d851286f2cf248549fc81f8296784c08" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:54.424371  8728 raft_consensus.cc:3060] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:54.425390  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:54.426281  8731 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
W20260709 13:04:54.430140  8696 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:54.430812  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.433133 16558 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.433941 16558 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.440869 16573 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.441823 16573 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.450980 16573 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:54.442617 16558 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.431892 16283 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:54.452860 16283 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.453267  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:54.452857 16560 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=1 applied=17 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:04:54.454228  8731 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:54.455660 16560 tablet_bootstrap.cc:492] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:04:54.455899  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.453534 16283 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:54.459324 16558 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:54.459833 16558 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.460222 16558 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:54.460685 16576 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.461556 16576 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.464201 16576 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:54.457321 16560 ts_tablet_manager.cc:1403] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 0.483s	user 0.082s	sys 0.097s
W20260709 13:04:54.476184  8696 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.487339 16283 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.490890  8728 raft_consensus.cc:2468] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:54.496510 16283 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:54.498195  8622 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.498898  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96614bb40dc54bb2bdcff8f9d5476bb1" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:54.503232  8731 raft_consensus.cc:3060] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:54.504041  9325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:54.513605  8625 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:54.518391 16541 raft_consensus.cc:2804] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:54.519343 16541 raft_consensus.cc:697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:54.520831 16541 consensus_queue.cc:237] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.522147 16575 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.522869 16575 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.518391 16577 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.525837 16577 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.526394 16575 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:54.528539 16574 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.529250 16574 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.533588 16574 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:54.534233  9294 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.535698 16577 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.538079  8697 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:54.539528 16576 raft_consensus.cc:2749] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:54.541314  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:54.543166  9326 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:54.543962  8654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:54.544862  8654 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 18.
I20260709 13:04:54.546120  9326 raft_consensus.cc:1275] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:04:54.549670  8731 raft_consensus.cc:2468] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 22.
I20260709 13:04:54.551173  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:54.551908  9326 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 19.
I20260709 13:04:54.558386  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:54.556725  8625 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.560911 16283 raft_consensus.cc:2804] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:54.561583 16283 raft_consensus.cc:697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:54.573712 16549 consensus_queue.cc:1048] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 13:04:54.578717 16558 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.589251  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.590492 16559 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:54.590981 16559 raft_consensus.cc:493] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.591405 16559 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:54.603787  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:54.604641  8732 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
W20260709 13:04:54.613204  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 44658.6us
I20260709 13:04:54.570199 16574 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:54.617316 16283 consensus_queue.cc:237] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.623908  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bab857d69b5f4bc480340a9324923bc4" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:54.630188  8732 raft_consensus.cc:3060] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:54.624631 16558 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:54.624778  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:04:54.625713  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:54.648736 16548 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:54.649317 16548 raft_consensus.cc:493] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.649839 16548 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:54.653029  8732 raft_consensus.cc:2468] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:54.655611  8625 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.654134 16541 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.658684 16541 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.659516 16558 raft_consensus.cc:2804] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:54.660468 16558 raft_consensus.cc:697] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:54.661715 16541 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:54.661767 16558 consensus_queue.cc:237] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.632511 16574 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.680172 16574 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:54.687263  8475 catalog_manager.cc:5697] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 20 to 21, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 21 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:54.688256 16559 raft_consensus.cc:515] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.698134 16559 leader_election.cc:290] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:54.722297  8622 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:54.723462  8622 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.688750 16560 raft_consensus.cc:359] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
W20260709 13:04:54.736886  7481 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 8605)
I20260709 13:04:54.731002 16560 raft_consensus.cc:740] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:04:54.738247 16560 consensus_queue.cc:260] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:04:54.742002 16560 ts_tablet_manager.cc:1434] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.269s	user 0.000s	sys 0.010s
W20260709 13:04:54.746816  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.749186 16576 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.730508 16559 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.749846 16576 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:54.748286  8696 leader_election.cc:336] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.750808 16559 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.770835 16582 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.772294 16582 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.774549 16582 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.765604 16548 raft_consensus.cc:515] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:54.794692  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 122087us
I20260709 13:04:54.795523  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:54.796452 16548 leader_election.cc:290] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.797200 16581 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:54.797835 16581 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.798262 16581 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:54.798786  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 77169us
W20260709 13:04:54.799059  8622 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:54.799867  8622 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:54.801568  8622 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.804872 16573 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.805642  8742 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-121ce91dbce74f019e528db7b7062a86 received from {username='slave'} at 127.0.0.1:42702
I20260709 13:04:54.808310  8742 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-121ce91dbce74f019e528db7b7062a86 on tablet 121ce91dbce74f019e528db7b7062a86 with peer 453e644bad5d410ab9be85b353e28dc2
I20260709 13:04:54.809852  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ed4dfb6b4d146c3bb0c1c3677d1b39b" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:54.810575  8731 raft_consensus.cc:3060] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:54.810642  8696 leader_election.cc:336] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:54.811349  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:54.810957 16573 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:54.811887  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:54.815402 16576 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:54.816066  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:54.816828 16577 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.817639 16577 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.816875  8732 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 16.
I20260709 13:04:54.819752  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:04:54.820465  8728 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:54.821542  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.822430 16582 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:54.822794 16582 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.823170 16582 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:54.825521 16577 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:54.853291  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:54.853850 16559 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:54.829195  8731 raft_consensus.cc:2468] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 22.
I20260709 13:04:54.841732  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.848290  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6c36d18684a44799bfb89a6d20e4fd7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:54.845562  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:04:54.866688  8655 raft_consensus.cc:3060] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:54.866858  9322 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:54.845534  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:54.868283  8651 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 20.
W20260709 13:04:54.846618  8696 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.869860  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
W20260709 13:04:54.831154  9294 leader_election.cc:336] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:54.872269  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.872622 16572 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:54.873803 16572 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.874271 16572 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:54.829524 16573 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:04:54.881186  8696 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.840577 16582 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.883498  8652 raft_consensus.cc:2393] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 17: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:54.885756 16587 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:54.885831  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b93c5bfc29f4210b1fa8340091624eb" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:54.886476 16587 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.886636  8731 raft_consensus.cc:3060] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:54.886991 16587 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:54.861424  8622 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:54.894913  8731 raft_consensus.cc:2468] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 17.
I20260709 13:04:54.899204  8625 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.900961 16558 raft_consensus.cc:2804] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:54.901540 16558 raft_consensus.cc:697] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:54.904179 16558 consensus_queue.cc:237] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:54.906929 16562 log.cc:927] Time spent T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 13:04:54.895879 16582 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.896548  9297 leader_election.cc:304] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:54.913172 16554 raft_consensus.cc:2804] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:54.913898 16554 raft_consensus.cc:697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:54.914915 16554 consensus_queue.cc:237] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.917104 16572 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:04:54.921387  8731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:04:54.921921 16572 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:04:54.922173  8731 raft_consensus.cc:3060] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:54.923065 16581 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.926628  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:54.928444 16573 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:54.928966 16573 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a0d703ee18464475a290425976246555)
I20260709 13:04:54.929879 16573 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:54.911849  8622 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:54.942143  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:54.951486  8652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b063565a0d8244cabf980570d2fc8a38" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:54.954809  8652 raft_consensus.cc:2393] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 21: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:54.925235 16581 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:54.955737  8655 raft_consensus.cc:2468] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:54.956848  9322 raft_consensus.cc:1217] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEARNER]: Deduplicated request from leader. Original: 22.16->[22.17-22.17]   Dedup: 22.17->[]
I20260709 13:04:54.961531  8731 raft_consensus.cc:2468] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 21.
I20260709 13:04:54.952821  8698 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:54.964900  8625 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:54.981635 16352 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:54.983999 16573 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:54.988983  8731 raft_consensus.cc:1275] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:04:54.990433 16573 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:54.994956 16283 consensus_queue.cc:1048] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:04:55.000069  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 56684.4us
I20260709 13:04:54.992744 16581 raft_consensus.cc:2804] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:55.002697  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e8a9411169f407cba1b44443edc9840" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:55.022578 16581 raft_consensus.cc:697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:55.022921  9323 raft_consensus.cc:3060] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:55.023774 16581 consensus_queue.cc:237] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:55.008539  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:55.015913  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:54.970402 16574 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:55.028205 16574 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:04:55.029408  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266236cfcd824409b757505321b36d81" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:55.030023  9326 raft_consensus.cc:3060] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:55.036607 16587 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:55.036967  8476 catalog_manager.cc:5697] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 21 to 22, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 22 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:55.039673  9326 raft_consensus.cc:2468] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
I20260709 13:04:55.041664  8697 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:04:55.042689 16578 raft_consensus.cc:2804] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:04:55.043318  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:55.044782  9323 raft_consensus.cc:2468] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a2b9c938d94a4fa258e068ea054d84 in term 20.
I20260709 13:04:55.043183 16578 raft_consensus.cc:697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:55.049252 16578 consensus_queue.cc:237] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:55.062755  8696 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:55.089756  8655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:55.090839  8655 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 453e644bad5d410ab9be85b353e28dc2 in current term 21: Already voted for candidate d0a2b9c938d94a4fa258e068ea054d84 in this term.
I20260709 13:04:55.064982 16587 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:55.097874  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:55.105360 16554 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:04:55.120741  8622 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:55.126055  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.126753  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.130251  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.130723  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:55.131912 16562 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=1 applied=14 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:04:55.132985 16562 tablet_bootstrap.cc:492] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: Bootstrap complete.
I20260709 13:04:55.139351  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:04:55.142895  8698 leader_election.cc:304] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:55.144670  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "d0a2b9c938d94a4fa258e068ea054d84" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:04:55.145494  9323 raft_consensus.cc:2393] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a2b9c938d94a4fa258e068ea054d84 in current term 21: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
I20260709 13:04:55.145974  8732 raft_consensus.cc:1275] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:55.146365 16352 raft_consensus.cc:2804] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:55.146842 16352 raft_consensus.cc:697] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:04:55.146908 16549 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:55.152454 16352 consensus_queue.cc:237] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:04:55.153102  8622 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:55.153945  8622 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:55.154428  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.154891  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.155400  8622 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:55.155861  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:55.163867 16572 consensus_queue.cc:1048] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
W20260709 13:04:55.163836  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:55.180872  8623 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:55.195595 16572 raft_consensus.cc:2804] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:55.196223 16572 raft_consensus.cc:697] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Becoming Leader. State: Replica: d0a2b9c938d94a4fa258e068ea054d84, State: Running, Role: LEADER
I20260709 13:04:55.199060  8728 raft_consensus.cc:1275] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:04:55.197160 16572 consensus_queue.cc:237] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:55.207044  8623 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a2b9c938d94a4fa258e068ea054d84; no voters: 453e644bad5d410ab9be85b353e28dc2, a0d703ee18464475a290425976246555
I20260709 13:04:55.203284 16562 ts_tablet_manager.cc:1403] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: Time spent bootstrapping tablet: real 1.076s	user 0.059s	sys 0.035s
I20260709 13:04:55.209898 16581 raft_consensus.cc:2749] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:55.212055 16571 consensus_queue.cc:1048] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.014s
I20260709 13:04:55.245193 16562 raft_consensus.cc:359] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:04:55.246199 16562 raft_consensus.cc:740] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEARNER]: Becoming Follower/Learner. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Initialized, Role: LEARNER
I20260709 13:04:55.246901 16562 consensus_queue.cc:260] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: NON_VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: true } }
I20260709 13:04:55.249009 16562 ts_tablet_manager.cc:1434] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.000s
W20260709 13:04:55.268575  8622 consensus_peers.cc:597] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:55.300765  8622 consensus_peers.cc:597] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:55.301604  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:55.309875  8742 tablet_copy_service.cc:342] P 1dc41db601fe4b91aa6d461b504da0e7: Request end of tablet copy session 453e644bad5d410ab9be85b353e28dc2-7778779e8695473fa6249308e7a4fa1e received from {username='slave'} at 127.0.0.1:42702
I20260709 13:04:55.310515  8742 tablet_copy_service.cc:434] P 1dc41db601fe4b91aa6d461b504da0e7: ending tablet copy session 453e644bad5d410ab9be85b353e28dc2-7778779e8695473fa6249308e7a4fa1e on tablet 7778779e8695473fa6249308e7a4fa1e with peer 453e644bad5d410ab9be85b353e28dc2
W20260709 13:04:55.320075  8622 consensus_peers.cc:597] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:55.337205  8476 catalog_manager.cc:5697] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 18 to 19, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 19 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:55.339190  8476 catalog_manager.cc:5697] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 18 to 19, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 19 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:55.340822  8476 catalog_manager.cc:5697] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 19 to 20, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 20 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:55.342371  8476 catalog_manager.cc:5697] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 21 to 22, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 22 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } }
I20260709 13:04:55.354194 16576 raft_consensus.cc:1064] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
W20260709 13:04:55.368286  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:55.376363 16576 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 15, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:55.377887  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.381850  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:55.394164  8475 catalog_manager.cc:5697] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 16 to 17, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 17 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:55.399417  8475 catalog_manager.cc:5697] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 20 to 21, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 21 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:55.410742  9331 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:04:55.412165  8654 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:04:55.413065 16549 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 13:04:55.421474 16352 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 13:04:55.401260  8475 catalog_manager.cc:5697] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: term changed from 19 to 20, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66). New cstate: current_term: 20 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:55.426385  9323 raft_consensus.cc:1217] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEARNER]: Deduplicated request from leader. Original: 21.13->[21.14-21.14]   Dedup: 21.14->[]
W20260709 13:04:55.454025  8696 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:55.458712  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:55.472642  8728 raft_consensus.cc:1275] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:55.474702 16587 consensus_queue.cc:1048] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:55.487614 16352 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Committing config change with OpId 22.18: config changed from index 17 to 18, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:55.508364  8654 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Committing config change with OpId 22.18: config changed from index 17 to 18, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:55.521265  8622 consensus_peers.cc:597] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:55.521827  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.524802  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:55.541267  9331 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Committing config change with OpId 22.18: config changed from index 17 to 18, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:55.561519  8475 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 17 to 18, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 22 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:55.575177  8622 consensus_peers.cc:597] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:55.579852  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:55.590845  8731 consensus_queue.cc:237] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:55.596088  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:55.601243  8728 raft_consensus.cc:1275] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:55.604091 16558 consensus_queue.cc:1048] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:55.631970  9331 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 22 index: 19. (index mismatch)
I20260709 13:04:55.640348 16549 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 13:04:55.645618  9331 raft_consensus.cc:1275] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:04:55.652490  8652 raft_consensus.cc:1275] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 22 index: 19. (index mismatch)
I20260709 13:04:55.656659  8731 raft_consensus.cc:1275] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:55.656965 16352 consensus_queue.cc:1048] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:04:55.658191 16572 consensus_queue.cc:1048] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:55.691891 16578 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Committing config change with OpId 22.19: config changed from index 18 to 19, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:55.697659 16559 consensus_queue.cc:1048] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:55.702535  8652 raft_consensus.cc:1275] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 13:04:55.705137  8622 consensus_peers.cc:597] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:55.707120 16352 consensus_queue.cc:1048] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:55.708784  8652 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Committing config change with OpId 22.19: config changed from index 18 to 19, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:55.724761  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:04:55.729043  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 121ce91dbce74f019e528db7b7062a86 with cas_config_opid_index 18: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 13:04:55.737670  7481 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 8605)
I20260709 13:04:55.749640  9323 raft_consensus.cc:2955] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Committing config change with OpId 22.19: config changed from index 18 to 19, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:55.800640  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:55.787048  8476 catalog_manager.cc:5697] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 reported cstate change: config changed from index 18 to 19, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 22 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:55.854933  9323 raft_consensus.cc:1275] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:04:55.861784 16594 consensus_queue.cc:1048] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:04:55.899374  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.914054  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.936635  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:55.997777  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 121ce91dbce74f019e528db7b7062a86: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:56.002013  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:56.022862 16578 raft_consensus.cc:1064] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: attempting to promote NON_VOTER 453e644bad5d410ab9be85b353e28dc2 to VOTER
I20260709 13:04:56.026647 16578 consensus_queue.cc:237] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 12, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:56.007856  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:56.046741  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:56.057518  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:04:56.078697  9323 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEARNER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:56.081754  8651 raft_consensus.cc:1275] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:56.083391 16578 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:56.085798 16549 consensus_queue.cc:1048] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 13:04:56.127544  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:56.152526 16549 raft_consensus.cc:2955] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Committing config change with OpId 21.15: config changed from index 14 to 15, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:56.160983  9323 raft_consensus.cc:2955] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Committing config change with OpId 21.15: config changed from index 14 to 15, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:56.182370  8651 raft_consensus.cc:2955] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Committing config change with OpId 21.15: config changed from index 14 to 15, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:56.199342  8476 catalog_manager.cc:5697] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 14 to 15, 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68) changed from NON_VOTER to VOTER. New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:56.345788  8622 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:56.393532  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:56.398162  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:56.483436  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:56.738160  7481 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 8605)
W20260709 13:04:56.750991  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:56.762435  8696 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:56.884876  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:04:56.999691 16587 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.000344 16587 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:57.002738 16587 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:04:57.007598  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:04:57.011927  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84" is_pre_election: true
I20260709 13:04:57.012933  8651 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 21.
I20260709 13:04:57.038861  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: 
I20260709 13:04:57.044435 16592 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:57.045056 16592 raft_consensus.cc:493] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:57.045534 16592 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:57.067763  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:57.109880  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 71089.3us
W20260709 13:04:57.146688  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:04:57.154376 16592 raft_consensus.cc:515] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:57.155676  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.178573  9294 leader_election.cc:336] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:57.182192  8651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98c31f9999e84270898dfb87c21cbc7c" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "d0a2b9c938d94a4fa258e068ea054d84"
I20260709 13:04:57.182881  8651 raft_consensus.cc:3060] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:57.200008 16592 leader_election.cc:290] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Requested vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:04:57.201661  8651 raft_consensus.cc:2468] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 22.
I20260709 13:04:57.215583  9296 leader_election.cc:304] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84; no voters: a0d703ee18464475a290425976246555
I20260709 13:04:57.217350 16587 raft_consensus.cc:2804] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:57.217931 16587 raft_consensus.cc:697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:04:57.218858 16587 consensus_queue.cc:237] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:04:57.237370  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:04:57.273725  8476 catalog_manager.cc:5697] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 20 to 22, leader changed from a0d703ee18464475a290425976246555 (127.7.78.65) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 22 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:57.338153  8466 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 74936.4us
W20260709 13:04:57.347779  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 121ce91dbce74f019e528db7b7062a86: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:57.382298  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.417690  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.439553  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.494129  8622 consensus_peers.cc:597] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.517838  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.547438  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.556648  8622 consensus_peers.cc:597] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.558080  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.571846  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:04:57.631222  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.651849  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.654956  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:04:57.666581  8654 raft_consensus.cc:1275] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:57.670106 16592 consensus_queue.cc:1048] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:04:57.732982  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:57.738677  7481 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 8605)
W20260709 13:04:57.794922  8622 consensus_peers.cc:597] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.797369  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.803361  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.814793  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.821601  8622 consensus_peers.cc:597] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.853945  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.882964  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.917311  8622 consensus_peers.cc:597] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.926335  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.930815  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:04:57.936225  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.949944  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:57.970470  8622 consensus_peers.cc:597] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:57.971454  8654 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:57.995810  9323 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:57.998050  8728 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Refusing update from remote peer d0a2b9c938d94a4fa258e068ea054d84: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:57.999040 16638 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:58.003422 16594 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:58.032629 16594 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Committing config change with OpId 15.13: config changed from index 9 to 13, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:58.039678  8728 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Committing config change with OpId 15.13: config changed from index 9 to 13, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:58.058902  9323 raft_consensus.cc:2955] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Committing config change with OpId 15.13: config changed from index 9 to 13, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:58.079873  8476 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: config changed from index 9 to 13, VOTER a0d703ee18464475a290425976246555 (127.7.78.65) evicted. New cstate: current_term: 15 leader_uuid: "d0a2b9c938d94a4fa258e068ea054d84" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
W20260709 13:04:58.099754  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:58.104019  8728 consensus_queue.cc:237] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:04:58.105634  8468 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f4f350f118fc449fb61357528a6b14f7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 11)
I20260709 13:04:58.142499  9323 raft_consensus.cc:1275] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 13:04:58.144603 16549 consensus_queue.cc:1048] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 13:04:58.147939  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:58.182287 16549 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 LEADER]: Committing config change with OpId 23.14: config changed from index 4 to 14, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:58.187402  9331 raft_consensus.cc:2955] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Committing config change with OpId 23.14: config changed from index 4 to 14, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } } }
I20260709 13:04:58.219960  8469 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 081470ef13974fe497f88859ada43ffa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 14)
I20260709 13:04:58.234227  8474 catalog_manager.cc:5697] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: config changed from index 4 to 14, VOTER d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) evicted. New cstate: current_term: 23 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:58.242853  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:04:58.251951  8647 tablet_service.cc:1558] Processing DeleteTablet for tablet 081470ef13974fe497f88859ada43ffa with delete_type TABLET_DATA_TOMBSTONED (TS d0a2b9c938d94a4fa258e068ea054d84 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:38080
W20260709 13:04:58.280387  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:58.291841  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:04:58.314169 16649 tablet_replica.cc:333] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:04:58.317458 16649 raft_consensus.cc:2243] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:58.318369 16649 raft_consensus.cc:2272] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:58.324676 16649 ts_tablet_manager.cc:1916] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 13:04:58.356769  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f4f350f118fc449fb61357528a6b14f7: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:04:58.363013 16649 ts_tablet_manager.cc:1929] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 23.13
I20260709 13:04:58.363790 16649 log.cc:1199] T 081470ef13974fe497f88859ada43ffa P d0a2b9c938d94a4fa258e068ea054d84: Deleting WAL directory at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/wals/081470ef13974fe497f88859ada43ffa
I20260709 13:04:58.368866  8468 catalog_manager.cc:5028] TS d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): tablet 081470ef13974fe497f88859ada43ffa (table test-workload [id=a52f92b6e8d04877ae60579197efba09]) successfully deleted
W20260709 13:04:58.507596  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:58.555578  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:58.632877  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:04:58.687527  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:58.698768  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51471.9us
W20260709 13:04:58.716310  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:58.728783  8696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111) [suppressed 468 similar messages]
W20260709 13:04:58.739231  7481 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 8605)
W20260709 13:04:58.742020  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:58.798336  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:58.903951  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:04:59.209000  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 101441us
W20260709 13:04:59.217967  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:04:59.506330  8696 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:59.545121  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:04:59.548063  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f4f350f118fc449fb61357528a6b14f7: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:04:59.739811  7481 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 8605)
W20260709 13:04:59.753379  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:04:59.806604  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:04:59.867280  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:04:59.932498  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:04:59.943815  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:04:59.946525  8622 consensus_peers.cc:597] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:59.966840  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 48722.5us
W20260709 13:04:59.978035  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.028019  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:00.035804  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.065014  8622 consensus_peers.cc:597] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.099656  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.117360  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.127700  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.213714  8622 consensus_peers.cc:597] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.322139  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.354128  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.379235  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.385264  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:00.389418  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:00.390972  8622 consensus_peers.cc:597] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.425765  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.454103  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.566594  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 88954.1us
W20260709 13:05:00.569485  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 83010.8us
W20260709 13:05:00.570974  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.571476  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 93618.4us
W20260709 13:05:00.573544  8622 consensus_peers.cc:597] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.573995  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 74885.9us
W20260709 13:05:00.654050  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.681828  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.716826  8623 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 239381us
W20260709 13:05:00.717832  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:00.719769  8466 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 68656.5us
W20260709 13:05:00.740355  7481 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 8605)
W20260709 13:05:00.741178  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 121ce91dbce74f019e528db7b7062a86: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:00.751066  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.760232  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.787460  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 71421.9us
W20260709 13:05:00.806285  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.809805  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:00.845952  8622 consensus_peers.cc:597] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.992148  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:01.122812  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:01.142974  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:01.155249  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:01.193315  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:01.241943  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:01.272887  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:01.282359  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:01.381735  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:01.433604  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:01.741214  7481 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 8605)
W20260709 13:05:01.751968  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:01.919828  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 200268us
W20260709 13:05:02.299104  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:02.308339  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:02.377140  8622 consensus_peers.cc:597] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:02.458554  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:02.492826  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:02.593705  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:02.717972  8622 consensus_peers.cc:597] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:02.742270  7481 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 8605)
W20260709 13:05:02.744488  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:02.772398  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:02.778470  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:02.781384  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:02.801241  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:02.868705  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:02.901587  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:02.929633  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:02.931628  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f4f350f118fc449fb61357528a6b14f7: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:02.961633  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080025bdc0 after lost signal to thread 7484
W20260709 13:05:02.962966  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001c3880 after lost signal to thread 8516
W20260709 13:05:02.964499  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08003bd880 after lost signal to thread 8683
W20260709 13:05:02.967655  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001c8860 after lost signal to thread 8757
W20260709 13:05:02.969142  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080014b380 after lost signal to thread 9355
W20260709 13:05:02.988633  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 47767.8us
W20260709 13:05:03.001211  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.053946  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.070330  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 48776.3us
W20260709 13:05:03.077214  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.077977  8696 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.180979  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 104299us
W20260709 13:05:03.631289  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.636364  8622 consensus_peers.cc:597] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.636912  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.637589  8622 consensus_peers.cc:597] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.638245  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.682092  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.688951  8622 consensus_peers.cc:597] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.708209  8622 consensus_peers.cc:597] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.715515  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.730908  9294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111) [suppressed 398 similar messages]
W20260709 13:05:03.742831  7481 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 8605)
W20260709 13:05:03.788811  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:03.808300  8698 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 44153.6us
W20260709 13:05:03.895805  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.906477  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.911180  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.912531  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.913130  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.913852  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.920574  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:03.953490  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:04.050472  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:04.178529  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:04.267155  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:04.358201  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:04.368261  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:04.517297  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:04.586772  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 57141.3us
W20260709 13:05:04.596426  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:04.621614  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:04.635619  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08000db2a0 after lost signal to thread 7484
W20260709 13:05:04.637797  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08003f7540 after lost signal to thread 8516
W20260709 13:05:04.640440  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08002f48e0 after lost signal to thread 8683
W20260709 13:05:04.646607  9352 debug-util.cc:398] Leaking SignalData structure 0x7b0800296320 after lost signal to thread 8757
W20260709 13:05:04.649027  9352 debug-util.cc:398] Leaking SignalData structure 0x7b080040cf20 after lost signal to thread 9355
W20260709 13:05:04.720293  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:04.743554  7481 thread.cc:527] Waited for 11000ms trying to join with diag-logger (tid 8605)
W20260709 13:05:04.829101  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:04.853339  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:04.923585  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:04.929966  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 121ce91dbce74f019e528db7b7062a86: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:04.991557  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.070782  8622 consensus_peers.cc:597] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.267872  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.287482  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.341015  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.347143  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:05.508030  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.517935  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.563118  8696 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.633903  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.644887  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.649340  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 51068.3us
W20260709 13:05:05.683801  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.744133  7481 thread.cc:527] Waited for 12000ms trying to join with diag-logger (tid 8605)
W20260709 13:05:05.834204  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 60431.5us
W20260709 13:05:05.881269  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.884260  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.955914  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.965507  8622 consensus_peers.cc:597] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.970667  8622 consensus_peers.cc:597] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.971524  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.973480  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:05.975481  8622 consensus_peers.cc:597] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.976080  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.001879  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.065541  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.117888  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 52512.1us
W20260709 13:05:06.181938  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 48007.1us
W20260709 13:05:06.190626  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.245833  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 55266us
W20260709 13:05:06.255632  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.256340  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.303898  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 51177.6us
W20260709 13:05:06.328740  8622 consensus_peers.cc:597] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:06.382769  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 54066.9us
W20260709 13:05:06.456000  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.498148  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.501142  8622 consensus_peers.cc:597] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:06.575745  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:06.585449  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.642071  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.680025  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:06.744714  7481 thread.cc:527] Waited for 13000ms trying to join with diag-logger (tid 8605)
W20260709 13:05:06.909827  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:07.006747  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:07.022815  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:07.071848  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:07.092352  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:07.096333  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f4f350f118fc449fb61357528a6b14f7: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:07.184311  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:07.201305  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:07.263123  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:07.342515  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:07.348086  8622 consensus_peers.cc:597] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:07.362951  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:07.504884  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:07.615360  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:07.684602  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:07.716562  8622 consensus_peers.cc:597] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:07.796221  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:07.868113  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 118876us
W20260709 13:05:07.900202  8625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 78414.4us
W20260709 13:05:07.947778  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 71124.7us
W20260709 13:05:08.140502  8622 consensus_peers.cc:597] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.244123  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.498288  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 254258us
W20260709 13:05:08.427131  8622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 173951us
W20260709 13:05:08.513769  8622 consensus_peers.cc:597] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.534777  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.543844  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.547874  8622 consensus_peers.cc:597] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.548727  8622 consensus_peers.cc:597] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.581331  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.585100  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.587883  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.592252  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.582559  8622 consensus_peers.cc:597] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.599524  8622 consensus_peers.cc:597] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.600104  8622 consensus_peers.cc:597] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.600996  8622 consensus_peers.cc:597] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.608364  8622 consensus_peers.cc:597] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.627058  8696 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.651906  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.676151  8622 consensus_peers.cc:597] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.676855  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.690150  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:08.725448  8622 consensus_peers.cc:597] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.740082  8622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111) [suppressed 453 similar messages]
W20260709 13:05:08.868804  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:08.864391  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
W20260709 13:05:09.010567  8622 consensus_peers.cc:597] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:09.067344  8622 consensus_peers.cc:597] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:09.181800  8622 consensus_peers.cc:597] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:09.219240  8622 consensus_peers.cc:597] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:09.305389  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08003d7140 after lost signal to thread 7484
W20260709 13:05:09.306574  8513 debug-util.cc:398] Leaking SignalData structure 0x7b080032a780 after lost signal to thread 8516
W20260709 13:05:09.308144  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08003650e0 after lost signal to thread 8683
W20260709 13:05:09.309345  8513 debug-util.cc:398] Leaking SignalData structure 0x7b08003281e0 after lost signal to thread 8757
W20260709 13:05:09.310951  8513 debug-util.cc:398] Leaking SignalData structure 0x7b0800320a20 after lost signal to thread 9355
W20260709 13:05:09.325503  8622 consensus_peers.cc:597] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:09.421305  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:09.473836  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
W20260709 13:05:09.525489  9296 consensus_peers.cc:597] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.547435  8698 consensus_peers.cc:597] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.548408  8698 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.557936  8698 consensus_peers.cc:597] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.558583  8698 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.559156  8698 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.566751  8622 consensus_peers.cc:597] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:09.640390  8698 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.658357  8698 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.664439  8698 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.666270  8698 consensus_peers.cc:597] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.678092  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:09.680465  9296 consensus_peers.cc:597] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.685804  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:09.709010  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:09.712654  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:09.714848  8698 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.741914  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:09.771461  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:05:09.777503  7481 tablet_replica.cc:333] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.779532  7481 raft_consensus.cc:2243] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:09.780586  7481 raft_consensus.cc:2272] T 90d5e31850d64b9eb8c754954c889846 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.783703  7481 tablet_replica.cc:333] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.784992  7481 raft_consensus.cc:2243] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Raft consensus shutting down.
I20260709 13:05:09.795609  7481 raft_consensus.cc:2272] T 4d18196b7e714cb6a79ec64ea3d65803 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:09.803674  8698 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.820667  7481 tablet_replica.cc:333] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.821729  7481 raft_consensus.cc:2243] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Raft consensus shutting down.
W20260709 13:05:09.825613  8698 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.826188  7481 raft_consensus.cc:2272] T 96cbbc16390646a9875b38f1b174f981 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.829306  7481 tablet_replica.cc:333] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
W20260709 13:05:09.830240  9296 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.834872  8622 consensus_peers.cc:597] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:09.847574  7481 raft_consensus.cc:2243] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:09.848589  7481 raft_consensus.cc:2272] T 652ad3041b404044aaf40f77c1be3a32 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:09.854101  8698 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.855125  7481 tablet_replica.cc:333] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.856048  7481 raft_consensus.cc:2243] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Raft consensus shutting down.
W20260709 13:05:09.861581  9296 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.857211  7481 raft_consensus.cc:2272] T 7eaeef2720084b6aac897de8d33f6921 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.873087  7481 tablet_replica.cc:333] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.880348  7481 raft_consensus.cc:2243] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:09.890928  7481 raft_consensus.cc:2272] T 9e8a9411169f407cba1b44443edc9840 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.893873  7481 tablet_replica.cc:333] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
W20260709 13:05:09.896730  8698 consensus_peers.cc:597] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.906777  7481 raft_consensus.cc:2243] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:09.907723  7481 raft_consensus.cc:2272] T 9413ba3105764883ae0d7597a86ad605 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.910737  7481 tablet_replica.cc:333] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.915340  7481 raft_consensus.cc:2243] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus shutting down.
W20260709 13:05:09.946527  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:09.949359  7481 raft_consensus.cc:2272] T dd0fc940d95d4b118d3e2564ed48b09e P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.952256  7481 tablet_replica.cc:333] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.958927  7481 raft_consensus.cc:2243] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:09.960934  7481 raft_consensus.cc:2272] T d851286f2cf248549fc81f8296784c08 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.963848  7481 tablet_replica.cc:333] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:09.965813  7481 raft_consensus.cc:2243] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 LEADER]: Raft consensus shutting down.
W20260709 13:05:09.967175  8698 consensus_peers.cc:597] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.972486  7481 raft_consensus.cc:2272] T 9f9dd5b88e0647f284658784381e1893 P d0a2b9c938d94a4fa258e068ea054d84 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:09.982561  7481 tablet_replica.cc:333] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
W20260709 13:05:09.983457  8698 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:09.990109  8698 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.990221  7481 raft_consensus.cc:2243] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:09.991724  7481 raft_consensus.cc:2272] T 96614bb40dc54bb2bdcff8f9d5476bb1 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:10.003754  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:10.007265  7481 tablet_replica.cc:333] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.008226  7481 raft_consensus.cc:2243] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.009505  7481 raft_consensus.cc:2272] T 96ad7a91a5c14870885cd58a34393f82 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.015686  7481 tablet_replica.cc:333] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.020248  7481 raft_consensus.cc:2243] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.021350  7481 raft_consensus.cc:2272] T f4f350f118fc449fb61357528a6b14f7 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.034509  7481 tablet_replica.cc:333] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.035547  7481 raft_consensus.cc:2243] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.036589  7481 raft_consensus.cc:2272] T cde36bf9fcc5409e8a00b747fbfbfb06 P d0a2b9c938d94a4fa258e068ea054d84 [term 16 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:10.039029  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:05:10.039353  7481 tablet_replica.cc:333] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.040228  7481 raft_consensus.cc:2243] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.041453  7481 raft_consensus.cc:2272] T 162db4080f654c9fbd8c23faf68f6df3 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.044446  7481 tablet_replica.cc:333] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.047940  7481 raft_consensus.cc:2243] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.058450  7481 raft_consensus.cc:2272] T bab857d69b5f4bc480340a9324923bc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.062338  7481 tablet_replica.cc:333] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.063246  7481 raft_consensus.cc:2243] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.064242  7481 raft_consensus.cc:2272] T bf63362bb9b4409299528319b7472b04 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.068156  7481 tablet_replica.cc:333] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.068996  7481 raft_consensus.cc:2243] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.075996  7481 raft_consensus.cc:2272] T 294596405ca140769bf30fa7ee1e4820 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.078902  7481 tablet_replica.cc:333] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.079803  7481 raft_consensus.cc:2243] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.080904  7481 raft_consensus.cc:2272] T 7778779e8695473fa6249308e7a4fa1e P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:10.084131  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:10.091595  7481 tablet_replica.cc:333] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.092522  7481 raft_consensus.cc:2243] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.093268  7481 raft_consensus.cc:2272] T a6c36d18684a44799bfb89a6d20e4fd7 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.095920  7481 tablet_replica.cc:333] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.106083  7481 raft_consensus.cc:2243] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.107542  7481 raft_consensus.cc:2272] T 04399390cf414886a3453981dc1f74aa P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.110489  7481 tablet_replica.cc:333] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.111579  7481 raft_consensus.cc:2243] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.112310  7481 raft_consensus.cc:2272] T 06f98a166dd945b39554b2f278c933ef P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.115302  7481 tablet_replica.cc:333] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.119904  7481 raft_consensus.cc:2243] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.120754  7481 raft_consensus.cc:2272] T b4572d49b22942189c377be6ba2293d6 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.123446  7481 tablet_replica.cc:333] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.129957  7481 raft_consensus.cc:2243] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.130828  7481 raft_consensus.cc:2272] T 01467ec5f29c4a728b24e55d1f5d2e94 P d0a2b9c938d94a4fa258e068ea054d84 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.133613  7481 tablet_replica.cc:333] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.136588  7481 raft_consensus.cc:2243] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.145638  7481 raft_consensus.cc:2272] T 121ce91dbce74f019e528db7b7062a86 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.148826  7481 tablet_replica.cc:333] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.150701  7481 raft_consensus.cc:2243] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.151890  7481 raft_consensus.cc:2272] T c3d38a98ca324ddbbb3ff25fb6ded965 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.154744  7481 tablet_replica.cc:333] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.159273  7481 raft_consensus.cc:2243] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.160427  7481 raft_consensus.cc:2272] T 3a6a2506bc27408a9b9113a0fa1f069f P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.168150  7481 tablet_replica.cc:333] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.168963  7481 raft_consensus.cc:2243] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.169729  7481 raft_consensus.cc:2272] T 2947646ebfc34bd7a5a55312f5812ee2 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.176608  7481 tablet_replica.cc:333] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.177476  7481 raft_consensus.cc:2243] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.178215  7481 raft_consensus.cc:2272] T a4f31f7ca2bf4416bec3531998325cc4 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.180785  7481 tablet_replica.cc:333] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.181524  7481 raft_consensus.cc:2243] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.182153  7481 raft_consensus.cc:2272] T f057517aa4a1496ab40134759e3d8b54 P d0a2b9c938d94a4fa258e068ea054d84 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.184962  7481 tablet_replica.cc:333] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
W20260709 13:05:10.193729  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:10.205945  7481 raft_consensus.cc:2243] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.206770  7481 raft_consensus.cc:2272] T ea2142382a5d4439915e07c70837c688 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.209615  7481 tablet_replica.cc:333] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.216012  7481 raft_consensus.cc:2243] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.217227  7481 raft_consensus.cc:2272] T 94c0917e28b9498a823f74b5f664a815 P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.221423  7481 tablet_replica.cc:333] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.227219  7481 raft_consensus.cc:2243] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.233884  7481 raft_consensus.cc:2272] T 77d6200607074a5ebd3e4ae470a2d6ed P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.236770  7481 tablet_replica.cc:333] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.237862  7481 raft_consensus.cc:2243] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.248140  7481 raft_consensus.cc:2272] T 4b06c2e2a7c644478ef688b47c11a07a P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:10.236483  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:10.251569  7481 tablet_replica.cc:333] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.262125  7481 raft_consensus.cc:2243] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.263509  7481 raft_consensus.cc:2272] T c6ad38ee7735403281a14a504308a8ea P d0a2b9c938d94a4fa258e068ea054d84 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.266726  7481 tablet_replica.cc:333] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.267597  7481 raft_consensus.cc:2243] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.268857  7481 raft_consensus.cc:2272] T d35f944458a44f6c82498c7fec026ec5 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.280984  7481 tablet_replica.cc:333] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.288574  7481 raft_consensus.cc:2243] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.289455  7481 raft_consensus.cc:2272] T 98c31f9999e84270898dfb87c21cbc7c P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.293187  7481 tablet_replica.cc:333] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.294420  7481 raft_consensus.cc:2243] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.306504  7481 raft_consensus.cc:2272] T d1f77a3af1d2472a857b0f7a57069d60 P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.310894  7481 tablet_replica.cc:333] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.311774  7481 raft_consensus.cc:2243] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.312696  7481 raft_consensus.cc:2272] T 8ffc6f04aee94f37b24daec463e6eb45 P d0a2b9c938d94a4fa258e068ea054d84 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.315814  7481 tablet_replica.cc:333] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.316741  7481 raft_consensus.cc:2243] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.320785  7481 raft_consensus.cc:2272] T a346f8961f774e87a3c75fa7e2053cb0 P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.324015  7481 tablet_replica.cc:333] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.331308  7481 raft_consensus.cc:2243] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.332672  7481 raft_consensus.cc:2272] T eed0f0ed87f74c1fb8320d38bceeac72 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.345633  7481 tablet_replica.cc:333] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.346541  7481 raft_consensus.cc:2243] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.348116  7481 raft_consensus.cc:2272] T b063565a0d8244cabf980570d2fc8a38 P d0a2b9c938d94a4fa258e068ea054d84 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.351243  7481 tablet_replica.cc:333] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.352092  7481 raft_consensus.cc:2243] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:10.352859  7481 raft_consensus.cc:2272] T 78613f64484447c487fa71abda1ec1ac P d0a2b9c938d94a4fa258e068ea054d84 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.355768  7481 tablet_replica.cc:333] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.366020  7481 raft_consensus.cc:2243] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.367542  7481 raft_consensus.cc:2272] T b387b386886048c8b66e5f499f02aab2 P d0a2b9c938d94a4fa258e068ea054d84 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.370702  7481 tablet_replica.cc:333] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.371515  7481 raft_consensus.cc:2243] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.372943  7481 raft_consensus.cc:2272] T 2adcf8a687b54060b644e8c691d8772f P d0a2b9c938d94a4fa258e068ea054d84 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.380906  7481 tablet_replica.cc:333] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.381817  7481 raft_consensus.cc:2243] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.383226  7481 raft_consensus.cc:2272] T eeef0c4282c74b3692ab849adb5ce58e P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:10.390668  7481 tablet_replica.cc:333] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84: stopping tablet replica
I20260709 13:05:10.391471  7481 raft_consensus.cc:2243] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:10.392820  7481 raft_consensus.cc:2272] T 6b93c5bfc29f4210b1fa8340091624eb P d0a2b9c938d94a4fa258e068ea054d84 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:10.410099  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:10.607633  8698 connection.cc:769] client connection to 127.7.78.66:39929 send error: Network error: sendmsg error: Broken pipe (error 32)
W20260709 13:05:10.631958  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:10.938812 16733 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:10.939502 16733 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:10.944372 16733 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:10.996025  8698 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:10.998621 16731 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.000228 16731 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.002621 16731 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:11.279335  9296 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.315114 16726 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.315773 16726 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.318130 16726 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.320122 16731 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.320722 16731 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.323356 16731 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:11.332343  8698 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.343401 16728 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.343637 16738 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.344110 16728 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.346390 16728 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.375680 16728 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.376379 16728 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.378912 16728 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.392040 16712 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.392709 16712 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.394821 16712 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.401413 16712 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.402056 16712 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.404320 16712 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.371428 16731 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.415145 16731 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:11.357687  9294 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:11.362099  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:11.366581  9296 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.419714  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:11.419921  9296 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:39929, user_credentials={real_user=slave}} blocked reactor thread for 53388.1us
W20260709 13:05:11.420356  9296 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.420670  9294 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 63017.3us
I20260709 13:05:11.407876 16736 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.422741 16736 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.344511 16738 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.409880 16735 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.425935 16735 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.412457 16734 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.428074 16734 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.430475 16740 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:05:11.419750  8696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:46779, user_credentials={real_user=slave}} blocked reactor thread for 57709.2us
W20260709 13:05:11.433641  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:11.434441  8696 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.434544 16736 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.434861  8696 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.435796 16735 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:05:11.436374 16735 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:11.437568 16738 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.438835 16734 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:05:11.424930  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:05:11.440910  8732 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:05:11.444507 16718 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.445053 16718 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.447530 16718 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:05:11.449551 16701 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.450078 16701 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.451180  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.452363 16701 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:11.452723  8698 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.452862 16701 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.453567 16701 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.452674 16738 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:05:11.455814 16701 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.455857 16738 raft_consensus.cc:493] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.456537 16738 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:05:11.458035  8698 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.417146 16731 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.461886  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:05:11.463567  9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:05:11.463805  9323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:05:11.468488 16741 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:05:11.469157  8698 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.469135 16741 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:11.470068  8698 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.470671  8698 leader_election.cc:336] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.471418 16741 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:11.477033 16701 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.477640 16701 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.479751 16701 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:11.481251  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:11.481606  9296 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.482182  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
W20260709 13:05:11.482338  8698 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.481063 16738 raft_consensus.cc:515] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:11.483223  8698 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.483985  9324 raft_consensus.cc:2393] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1dc41db601fe4b91aa6d461b504da0e7 in current term 18: Already voted for candidate 453e644bad5d410ab9be85b353e28dc2 in this term.
W20260709 13:05:11.483912  8698 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.482267  9294 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.484935  9294 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.487205 16741 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:11.482190  9296 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.489830  9294 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:11.495627  9296 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.497875 16738 leader_election.cc:290] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:11.498721  8697 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.502162 16736 raft_consensus.cc:2749] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:05:11.499300  9294 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.499537 16738 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:05:11.503342  9296 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.503538 16738 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:11.507965  9294 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.508810  9294 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.506492 16741 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:11.507099  8697 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.507889 16740 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
W20260709 13:05:11.504894  8698 leader_election.cc:336] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.506018  8696 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:11.510104 16741 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:05:11.510944 16738 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:11.507963 16739 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.511634 16739 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.513855 16739 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:11.514300  9296 leader_election.cc:336] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.514339 16736 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.515084 16736 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:11.515940  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:11.509714 16740 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.519976 16740 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.517292 16736 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:05:11.522826 16740 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.523046 16701 raft_consensus.cc:2749] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:05:11.521941  9294 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.523500 16740 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.523895  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.524667  8697 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: 453e644bad5d410ab9be85b353e28dc2, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:11.524535  8696 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.525377  8696 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.526088 16701 raft_consensus.cc:3060] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:05:11.526908 16740 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:11.526042  9296 leader_election.cc:336] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.527897  9296 leader_election.cc:336] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.529026  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.530201 16740 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:11.531294 16735 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:11.533329  9296 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.546605 16740 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:05:11.547508  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:05:11.548154  9324 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 19.
W20260709 13:05:11.549256  9294 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.554039  9294 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.554977 16740 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:05:11.556006  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162db4080f654c9fbd8c23faf68f6df3" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:05:11.556631  8698 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.551297  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:11.551227  9296 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.539927 16701 raft_consensus.cc:2749] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:05:11.557499  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:05:11.558152  8728 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 17.
I20260709 13:05:11.565656  8732 raft_consensus.cc:2468] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:05:11.557538  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.568576 16735 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:11.569545  8696 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.570787  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.570240  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:05:11.571691  8732 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 23.
I20260709 13:05:11.572204 16735 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:05:11.572970  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:11.573585  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:05:11.585675  8728 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 15.
I20260709 13:05:11.574651  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.587273  9297 leader_election.cc:304] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.588332 16740 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:05:11.588696 16740 raft_consensus.cc:493] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.589057 16740 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:05:11.593118 16739 raft_consensus.cc:2804] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:05:11.593711  8696 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.594128  8696 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.593719 16739 raft_consensus.cc:697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:05:11.595314 16739 consensus_queue.cc:237] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.595896  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.598702 16733 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:05:11.599083 16733 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.599421 16733 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:05:11.602335  8696 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:11.602654 16726 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:11.602926  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:11.603356  8696 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.603714  8696 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:11.604625  8696 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.604979  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.606391 16701 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:05:11.607133 16701 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:11.609979 16738 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.610644 16738 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.614156 16738 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.614976  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.616426  9297 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.617295 16741 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:05:11.617802 16741 raft_consensus.cc:493] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.618280 16741 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:05:11.619357  9294 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:11.620786  9296 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.624607 16728 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:05:11.625085 16728 raft_consensus.cc:493] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.625597 16728 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:05:11.625536  9296 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.630461 16731 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:11.632815  8696 leader_election.cc:336] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.633364  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:11.634738  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:11.635318 16736 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:11.652415 16741 raft_consensus.cc:515] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:11.668749  8696 consensus_peers.cc:597] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:11.655915 16733 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.639612 16740 raft_consensus.cc:515] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.664642 16739 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.671977 16739 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.673336  9324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ad38ee7735403281a14a504308a8ea" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:05:11.674134  9324 raft_consensus.cc:3060] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:05:11.675061  8476 catalog_manager.cc:5697] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 18, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:11.675912  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7" is_pre_election: true
I20260709 13:05:11.677791 16739 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:11.679361  9296 leader_election.cc:336] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.681038  8728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3d38a98ca324ddbbb3ff25fb6ded965" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:05:11.681665  8728 raft_consensus.cc:3060] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:05:11.680056 16735 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.683070 16735 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.685295 16735 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:05:11.685611  9296 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.687932  9296 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:11.685817  8698 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.691123 16739 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:11.665719 16728 raft_consensus.cc:515] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.692554  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2" is_pre_election: true
I20260709 13:05:11.693229  9322 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
I20260709 13:05:11.696708  8732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f350f118fc449fb61357528a6b14f7" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
W20260709 13:05:11.697055  9296 leader_election.cc:336] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.697428  8732 raft_consensus.cc:3060] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:05:11.698218  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: 
I20260709 13:05:11.699102 16735 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:05:11.699720 16735 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting leader election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.700183 16735 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:05:11.702445  9324 raft_consensus.cc:2468] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 20.
W20260709 13:05:11.704824  8698 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.709678 16739 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.710713 16739 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.714596 16739 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.715613 16740 leader_election.cc:290] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:11.717800 16741 leader_election.cc:290] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:11.718029  8724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96ad7a91a5c14870885cd58a34393f82" candidate_uuid: "453e644bad5d410ab9be85b353e28dc2" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "1dc41db601fe4b91aa6d461b504da0e7"
I20260709 13:05:11.718240 16728 leader_election.cc:290] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:11.720773  8728 raft_consensus.cc:2468] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 18.
I20260709 13:05:11.726720  8724 raft_consensus.cc:3060] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:11.721261 16733 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
W20260709 13:05:11.724512  9294 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.727880  9297 leader_election.cc:304] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.729686  8697 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.730779 16733 raft_consensus.cc:2804] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:05:11.731420 16728 raft_consensus.cc:2804] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:05:11.731948 16728 raft_consensus.cc:697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
W20260709 13:05:11.733166  9296 leader_election.cc:336] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.735241 16733 raft_consensus.cc:697] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
W20260709 13:05:11.736536  9296 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.737104  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.732949 16728 consensus_queue.cc:237] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.737941 16740 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:11.736351 16733 consensus_queue.cc:237] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.748415  8724 raft_consensus.cc:2468] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 24.
I20260709 13:05:11.749464 16735 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.750362  9297 leader_election.cc:304] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.751916 16741 raft_consensus.cc:2804] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:05:11.725243  8732 raft_consensus.cc:2468] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453e644bad5d410ab9be85b353e28dc2 in term 16.
I20260709 13:05:11.752444 16741 raft_consensus.cc:697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 24 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:05:11.753988  9297 leader_election.cc:304] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.756680 16739 raft_consensus.cc:2804] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:05:11.757330 16739 raft_consensus.cc:697] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Becoming Leader. State: Replica: 453e644bad5d410ab9be85b353e28dc2, State: Running, Role: LEADER
I20260709 13:05:11.757589 16741 consensus_queue.cc:237] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.758390 16739 consensus_queue.cc:237] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:11.768860  8698 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.769246  9296 consensus_peers.cc:597] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:11.770545 16735 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Requested vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68:33133)
I20260709 13:05:11.773437  9322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d35f944458a44f6c82498c7fec026ec5" candidate_uuid: "1dc41db601fe4b91aa6d461b504da0e7" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "453e644bad5d410ab9be85b353e28dc2"
I20260709 13:05:11.774341  9322 raft_consensus.cc:3060] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:05:11.790016  9322 raft_consensus.cc:2468] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dc41db601fe4b91aa6d461b504da0e7 in term 21.
I20260709 13:05:11.790665 16738 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.793444  8697 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7, 453e644bad5d410ab9be85b353e28dc2; no voters: d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.791672 16738 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.796837 16736 raft_consensus.cc:2804] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:05:11.797327 16736 raft_consensus.cc:697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Becoming Leader. State: Replica: 1dc41db601fe4b91aa6d461b504da0e7, State: Running, Role: LEADER
I20260709 13:05:11.798269 16738 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.798597 16736 consensus_queue.cc:237] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.814702 16701 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.815433 16701 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.818512 16701 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:11.821687 16701 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.822383 16701 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:11.825276 16701 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:11.837760  9294 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:11.839596  8696 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:11.841615  8698 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:11.849541  8696 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:11.849644  8698 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.851315 16739 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:11.851913 16739 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:11.852020  8696 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.853240 16726 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:11.854295 16739 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:11.854542  8696 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:11.855229  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.861029 16726 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:05:11.864266  9294 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:11.855404  9296 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.865784  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.865278  8474 catalog_manager.cc:5697] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 19 to 20, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 20 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:11.869031 16738 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:05:11.896003  9296 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:11.896526  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:11.902397 16738 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:11.920687  8754 debug-util.cc:398] Leaking SignalData structure 0x7b08001a8d40 after lost signal to thread 7484
W20260709 13:05:11.921944  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080029b2c0 after lost signal to thread 8516
W20260709 13:05:11.923404  8754 debug-util.cc:398] Leaking SignalData structure 0x7b0800153760 after lost signal to thread 8757
W20260709 13:05:11.924954  8754 debug-util.cc:398] Leaking SignalData structure 0x7b080023f7e0 after lost signal to thread 9355
W20260709 13:05:11.936378  7481 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 8680)
W20260709 13:05:11.943837  8698 consensus_peers.cc:597] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:11.952433  8476 catalog_manager.cc:5697] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 17 to 18, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 18 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:11.948855  8474 catalog_manager.cc:5697] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 reported cstate change: term changed from 20 to 21, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67). New cstate: current_term: 21 leader_uuid: "1dc41db601fe4b91aa6d461b504da0e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:11.954849  8476 catalog_manager.cc:5697] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 23 to 24, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 24 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:11.957703  8476 catalog_manager.cc:5697] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 reported cstate change: term changed from 15 to 16, leader changed from d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66) to 453e644bad5d410ab9be85b353e28dc2 (127.7.78.68). New cstate: current_term: 16 leader_uuid: "453e644bad5d410ab9be85b353e28dc2" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:05:11.968101  8698 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:11.997295  8698 consensus_peers.cc:597] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.020875  9296 consensus_peers.cc:597] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:12.073992  8698 consensus_peers.cc:597] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.078110  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:05:12.104871  8732 raft_consensus.cc:1275] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:05:12.106695 16741 consensus_queue.cc:1048] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:05:12.135905  9324 raft_consensus.cc:1275] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:05:12.140834  8724 raft_consensus.cc:1275] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:05:12.140618 16726 consensus_queue.cc:1048] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:05:12.143131 16741 consensus_queue.cc:1048] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 13:05:12.182157  8696 consensus_peers.cc:597] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:12.185636  9296 consensus_peers.cc:597] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:12.207763  8733 raft_consensus.cc:1275] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 13:05:12.208173  9322 raft_consensus.cc:1275] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Refusing update from remote peer 1dc41db601fe4b91aa6d461b504da0e7: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260709 13:05:12.220247 16726 consensus_queue.cc:1048] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 13:05:12.220950 16741 consensus_queue.cc:1048] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:05:12.359714  8698 consensus_peers.cc:597] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:12.366335  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:12.367128  8698 consensus_peers.cc:597] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:12.373840  9296 consensus_peers.cc:597] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:12.374435  9296 consensus_peers.cc:597] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.387950  9296 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:12.405062  9297 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37975, user_credentials={real_user=slave}} blocked reactor thread for 59379.3us
W20260709 13:05:12.411872  9352 debug-util.cc:398] Leaking SignalData structure 0x7b080005d800 after lost signal to thread 7484
I20260709 13:05:12.412179  8728 raft_consensus.cc:1275] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 24 FOLLOWER]: Refusing update from remote peer 453e644bad5d410ab9be85b353e28dc2: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 13:05:12.413091  9352 debug-util.cc:398] Leaking SignalData structure 0x7b0800196400 after lost signal to thread 8516
W20260709 13:05:12.416252  8698 consensus_peers.cc:597] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.416746  9352 debug-util.cc:398] Leaking SignalData structure 0x7b080039a2c0 after lost signal to thread 8757
W20260709 13:05:12.418329  9352 debug-util.cc:398] Leaking SignalData structure 0x7b08003b7800 after lost signal to thread 9355
W20260709 13:05:12.411841  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:12.428663  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:12.435590 16767 consensus_queue.cc:1048] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 13:05:12.479009  8698 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.516927  8698 consensus_peers.cc:597] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.535003  9296 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.541857  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:12.542593  8698 consensus_peers.cc:597] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.561046  8698 consensus_peers.cc:597] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.561568  8698 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.570160  8698 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.583168  8698 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.601679  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:12.655929  8698 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.693707  8698 consensus_peers.cc:597] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.719312  8698 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.828197  8698 consensus_peers.cc:597] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:12.855304  8696 consensus_peers.cc:597] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:12.874136  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:12.876989  9296 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:12.947249 16767 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:12.947788 16767 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:12.949966 16767 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:12.960055  9296 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:12.960817  9294 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:12.961264  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:12.962510 16767 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:05:12.996692  8696 consensus_peers.cc:597] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:13.035734  8696 consensus_peers.cc:597] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:13.052930 16733 raft_consensus.cc:493] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.053577 16733 raft_consensus.cc:515] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.055783 16733 leader_election.cc:290] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.066130  8696 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:13.067009  8698 leader_election.cc:336] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.067512  8698 leader_election.cc:304] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.068629 16733 raft_consensus.cc:2749] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:13.087670 16767 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.088372 16767 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:13.090984 16740 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.091718 16740 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:13.105904  9296 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.112041 16740 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:13.110392 16767 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.107774  9294 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:13.113232  9294 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.115083  9294 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.118884 16767 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:05:13.121430  9296 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.122164  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.123661 16767 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:05:13.134645  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:13.141113  8696 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:13.142364  8698 consensus_peers.cc:597] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:13.229653 16767 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.230352 16767 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:13.239838 16767 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.245379  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 121ce91dbce74f019e528db7b7062a86: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:13.243942  9296 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:13.240196  8696 consensus_peers.cc:597] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:13.252733  9294 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.253753  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.255314 16767 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:13.312469 16767 raft_consensus.cc:493] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.313088 16767 raft_consensus.cc:515] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:13.322865  9294 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.323433 16767 leader_election.cc:290] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.330267  9296 leader_election.cc:336] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.330909  9296 leader_election.cc:304] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.334982 16767 raft_consensus.cc:2749] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:13.350723 16766 raft_consensus.cc:493] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.351416 16766 raft_consensus.cc:515] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.357311 16733 raft_consensus.cc:493] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.357929 16733 raft_consensus.cc:515] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.360456 16733 leader_election.cc:290] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.365451  8698 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.381548 16766 leader_election.cc:290] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.383896  8698 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:13.383229  8696 leader_election.cc:336] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.384913  8696 leader_election.cc:304] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.385970 16772 raft_consensus.cc:2749] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:05:13.388731  8696 leader_election.cc:336] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.389194  8696 leader_election.cc:304] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.397060 16772 raft_consensus.cc:2749] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:13.404291 16767 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.405064 16767 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:13.407905 16767 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.414682  9294 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:13.433240  9296 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.445814  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.447711 16767 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:05:13.507328  8696 consensus_peers.cc:597] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:13.553287  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
I20260709 13:05:13.554591 16772 raft_consensus.cc:493] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.555106 16772 raft_consensus.cc:515] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.559096 16733 raft_consensus.cc:493] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.559656 16733 raft_consensus.cc:515] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
W20260709 13:05:13.561503  8698 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.563612 16733 leader_election.cc:290] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:13.557353 16772 leader_election.cc:290] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.581601  8698 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:13.584196  8696 leader_election.cc:336] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.587133  8696 leader_election.cc:304] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.587911 16772 raft_consensus.cc:2749] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:05:13.588845  8696 leader_election.cc:336] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.589232  8696 leader_election.cc:304] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.590689 16772 raft_consensus.cc:2749] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:13.591871 16772 raft_consensus.cc:493] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.592334 16772 raft_consensus.cc:515] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.594324 16772 leader_election.cc:290] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.599948  8696 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.611555 16772 raft_consensus.cc:493] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.612052 16772 raft_consensus.cc:515] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.614117 16772 leader_election.cc:290] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.625877  8696 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.625970 16740 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.628116 16740 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:13.631004 16740 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.627367  8698 leader_election.cc:336] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.632092  8698 leader_election.cc:304] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.632925 16772 raft_consensus.cc:2749] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:13.639670  9296 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:13.641260  9294 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.641741  9294 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.643158 16767 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:13.642729 16772 raft_consensus.cc:493] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.644958 16772 raft_consensus.cc:515] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.647161 16772 leader_election.cc:290] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.649712  8698 leader_election.cc:336] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.656982  8698 leader_election.cc:304] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.659675 16772 raft_consensus.cc:2749] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:13.662712  8696 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:13.679353  8698 leader_election.cc:336] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.681890  8698 leader_election.cc:304] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.684520 16772 raft_consensus.cc:2749] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:13.689262 16772 raft_consensus.cc:493] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.689989 16772 raft_consensus.cc:515] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.692400 16772 leader_election.cc:290] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:13.702595  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
W20260709 13:05:13.709926  8696 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.718758 16740 raft_consensus.cc:493] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.731616 16740 raft_consensus.cc:515] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:13.719429  9297 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.741191  9297 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 564 similar messages]
W20260709 13:05:13.743065  8698 leader_election.cc:336] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:13.741998  9297 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.745816  9297 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.753369  9294 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:13.755215  9296 leader_election.cc:336] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.755643  9296 leader_election.cc:304] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.748325 16740 leader_election.cc:290] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.759881  8696 consensus_peers.cc:597] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:13.758297 16741 raft_consensus.cc:2749] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:13.762961  8698 leader_election.cc:304] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:13.767879  9297 consensus_peers.cc:597] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:13.771211 16766 raft_consensus.cc:2749] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:05:13.825902 16773 raft_consensus.cc:493] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.830550 16773 raft_consensus.cc:515] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } }
I20260709 13:05:13.843158 16773 leader_election.cc:290] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:13.855685  8698 leader_election.cc:336] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:13.865692  8696 consensus_peers.cc:597] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:13.866303  8696 leader_election.cc:336] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.866765  8696 leader_election.cc:304] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dc41db601fe4b91aa6d461b504da0e7; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.867645 16773 raft_consensus.cc:2749] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:05:13.868971  8697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:33133, user_credentials={real_user=slave}} blocked reactor thread for 97879.1us
W20260709 13:05:13.872524  9297 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.889735  9297 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.891690  9297 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.896720  9297 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:13.902268  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:05:13.906288 16767 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:13.906949 16767 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:13.905649  7481 tablet_replica.cc:333] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.909868  7481 raft_consensus.cc:2243] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.910712  7481 raft_consensus.cc:2272] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.909132 16767 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:13.914125  7481 tablet_replica.cc:333] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
W20260709 13:05:13.915879  9294 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:13.916417  7481 raft_consensus.cc:2243] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:13.917654  7481 raft_consensus.cc:2272] T 90d5e31850d64b9eb8c754954c889846 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:13.920846  9297 consensus_peers.cc:597] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.921368  9296 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:13.922273  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:13.923592 16767 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:13.925799  8696 consensus_peers.cc:597] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:05:13.927047  7481 tablet_replica.cc:333] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.928018  7481 raft_consensus.cc:2243] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.928958  7481 raft_consensus.cc:2272] T 96cbbc16390646a9875b38f1b174f981 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:13.931902  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:05:13.932097  7481 tablet_replica.cc:333] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.933130  7481 raft_consensus.cc:2243] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.935343  7481 raft_consensus.cc:2272] T 7eaeef2720084b6aac897de8d33f6921 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.939904  7481 tablet_replica.cc:333] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.940915  7481 raft_consensus.cc:2243] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.941708  7481 raft_consensus.cc:2272] T 3ad2aa6046e746348b02a2571b7580f0 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.944319  7481 tablet_replica.cc:333] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.945133  7481 raft_consensus.cc:2243] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.945921  7481 raft_consensus.cc:2272] T 294596405ca140769bf30fa7ee1e4820 P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.950690  7481 tablet_replica.cc:333] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.951645  7481 raft_consensus.cc:2243] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:13.952900  7481 raft_consensus.cc:2272] T 9413ba3105764883ae0d7597a86ad605 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.955742  7481 tablet_replica.cc:333] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.956667  7481 raft_consensus.cc:2243] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:13.957930  7481 raft_consensus.cc:2272] T dd0fc940d95d4b118d3e2564ed48b09e P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.961174  7481 tablet_replica.cc:333] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.962141  7481 raft_consensus.cc:2243] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.963014  7481 raft_consensus.cc:2272] T d851286f2cf248549fc81f8296784c08 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:13.970357  9297 consensus_peers.cc:597] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:13.971338  8696 consensus_peers.cc:597] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:05:13.972570  7481 tablet_replica.cc:333] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.973539  7481 raft_consensus.cc:2243] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:13.975466  7481 raft_consensus.cc:2272] T 081470ef13974fe497f88859ada43ffa P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.978232  7481 tablet_replica.cc:333] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.979180  7481 raft_consensus.cc:2243] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.979889  7481 raft_consensus.cc:2272] T 96614bb40dc54bb2bdcff8f9d5476bb1 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:13.985769  8696 consensus_peers.cc:597] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:13.988327  7481 tablet_replica.cc:333] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.989313  7481 raft_consensus.cc:2243] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:13.990947  7481 raft_consensus.cc:2272] T 7778779e8695473fa6249308e7a4fa1e P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.993921  7481 tablet_replica.cc:333] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.994761  7481 raft_consensus.cc:2243] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:13.995409  7481 raft_consensus.cc:2272] T 162db4080f654c9fbd8c23faf68f6df3 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:13.997942  7481 tablet_replica.cc:333] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:13.998869  7481 raft_consensus.cc:2243] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:13.999963  7481 raft_consensus.cc:2272] T bf63362bb9b4409299528319b7472b04 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.004690  7481 tablet_replica.cc:333] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.005642  7481 raft_consensus.cc:2243] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.006484  7481 raft_consensus.cc:2272] T bab857d69b5f4bc480340a9324923bc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.009006  7481 tablet_replica.cc:333] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.010141  7481 raft_consensus.cc:2243] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.011461  7481 raft_consensus.cc:2272] T b4572d49b22942189c377be6ba2293d6 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.014127  7481 tablet_replica.cc:333] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.015030  7481 raft_consensus.cc:2243] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.016160  7481 raft_consensus.cc:2272] T ea2142382a5d4439915e07c70837c688 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.018942  7481 tablet_replica.cc:333] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.019739  7481 raft_consensus.cc:2243] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.020359  7481 raft_consensus.cc:2272] T 96ad7a91a5c14870885cd58a34393f82 P 1dc41db601fe4b91aa6d461b504da0e7 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.023128  7481 tablet_replica.cc:333] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.028019  7481 raft_consensus.cc:2243] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.028877  7481 raft_consensus.cc:2272] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.037695  7481 tablet_replica.cc:333] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.038647  7481 raft_consensus.cc:2243] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.039327  7481 raft_consensus.cc:2272] T f4f350f118fc449fb61357528a6b14f7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.041985  7481 tablet_replica.cc:333] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.042870  7481 raft_consensus.cc:2243] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.044142  7481 raft_consensus.cc:2272] T a6c36d18684a44799bfb89a6d20e4fd7 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.046968  7481 tablet_replica.cc:333] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.047860  7481 raft_consensus.cc:2243] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.049242  7481 raft_consensus.cc:2272] T 06f98a166dd945b39554b2f278c933ef P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.051867  7481 tablet_replica.cc:333] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.052721  7481 raft_consensus.cc:2243] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.054214  7481 raft_consensus.cc:2272] T a4f31f7ca2bf4416bec3531998325cc4 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.056813  7481 tablet_replica.cc:333] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.058063  7481 raft_consensus.cc:2243] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.059121  7481 raft_consensus.cc:2272] T f057517aa4a1496ab40134759e3d8b54 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.061858  7481 tablet_replica.cc:333] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.062700  7481 raft_consensus.cc:2243] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.063465  7481 raft_consensus.cc:2272] T 689ad8326d634dee8402f1352a76f0e2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.066161  7481 tablet_replica.cc:333] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.066987  7481 raft_consensus.cc:2243] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.068122  7481 raft_consensus.cc:2272] T 01467ec5f29c4a728b24e55d1f5d2e94 P 1dc41db601fe4b91aa6d461b504da0e7 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.070928  7481 tablet_replica.cc:333] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.071802  7481 raft_consensus.cc:2243] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.073014  7481 raft_consensus.cc:2272] T 121ce91dbce74f019e528db7b7062a86 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.075701  7481 tablet_replica.cc:333] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.076571  7481 raft_consensus.cc:2243] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.077539  7481 raft_consensus.cc:2272] T c3d38a98ca324ddbbb3ff25fb6ded965 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.079936  7481 tablet_replica.cc:333] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.080711  7481 raft_consensus.cc:2243] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.083499  7481 raft_consensus.cc:2272] T 2947646ebfc34bd7a5a55312f5812ee2 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.086216  7481 tablet_replica.cc:333] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.087118  7481 raft_consensus.cc:2243] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.088276  7481 raft_consensus.cc:2272] T 3a6a2506bc27408a9b9113a0fa1f069f P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.091135  7481 tablet_replica.cc:333] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.091939  7481 raft_consensus.cc:2243] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.092720  7481 raft_consensus.cc:2272] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.095589  7481 tablet_replica.cc:333] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.096438  7481 raft_consensus.cc:2243] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.097594  7481 raft_consensus.cc:2272] T 03f3a12327224113b1c5bf090e280a53 P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.100332  7481 tablet_replica.cc:333] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.101136  7481 raft_consensus.cc:2243] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Raft consensus shutting down.
W20260709 13:05:14.105497  9297 consensus_peers.cc:597] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:14.106125  7481 raft_consensus.cc:2272] T 652ad3041b404044aaf40f77c1be3a32 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.109150  7481 tablet_replica.cc:333] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.110172  7481 raft_consensus.cc:2243] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.111443  7481 raft_consensus.cc:2272] T c6ad38ee7735403281a14a504308a8ea P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.114491  7481 tablet_replica.cc:333] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.115633  7481 raft_consensus.cc:2243] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.118319  7481 raft_consensus.cc:2272] T 266236cfcd824409b757505321b36d81 P 1dc41db601fe4b91aa6d461b504da0e7 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.119462 16740 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:14.130482 16740 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:14.132691  7481 tablet_replica.cc:333] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.133759  7481 raft_consensus.cc:2243] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.134531  7481 raft_consensus.cc:2272] T 6b93c5bfc29f4210b1fa8340091624eb P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.132691 16740 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:14.141177  9294 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:14.141228  7481 tablet_replica.cc:333] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
W20260709 13:05:14.141906  9296 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:14.142351  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:14.142635  7481 raft_consensus.cc:2243] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.143018 16740 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:05:14.143491  7481 raft_consensus.cc:2272] T b063565a0d8244cabf980570d2fc8a38 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.146978  7481 tablet_replica.cc:333] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.147857  7481 raft_consensus.cc:2243] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.148542  7481 raft_consensus.cc:2272] T 77d6200607074a5ebd3e4ae470a2d6ed P 1dc41db601fe4b91aa6d461b504da0e7 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.151454  7481 tablet_replica.cc:333] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.152290  7481 raft_consensus.cc:2243] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.153025  7481 raft_consensus.cc:2272] T d1f77a3af1d2472a857b0f7a57069d60 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.155823  7481 tablet_replica.cc:333] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.156672  7481 raft_consensus.cc:2243] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.158145  7481 raft_consensus.cc:2272] T 9a389d19cf2c47d6b5873139d8cfad40 P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.161051  7481 tablet_replica.cc:333] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.161916  7481 raft_consensus.cc:2243] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.162544  7481 raft_consensus.cc:2272] T 78613f64484447c487fa71abda1ec1ac P 1dc41db601fe4b91aa6d461b504da0e7 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.165261  7481 tablet_replica.cc:333] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.166085  7481 raft_consensus.cc:2243] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.167249  7481 raft_consensus.cc:2272] T d35f944458a44f6c82498c7fec026ec5 P 1dc41db601fe4b91aa6d461b504da0e7 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.169896  7481 tablet_replica.cc:333] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.170684  7481 raft_consensus.cc:2243] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.171638  7481 raft_consensus.cc:2272] T 5559db905acf4ba0b48e9e3e87f49c1c P 1dc41db601fe4b91aa6d461b504da0e7 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.174456  7481 tablet_replica.cc:333] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.175236  7481 raft_consensus.cc:2243] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.176169  7481 raft_consensus.cc:2272] T a7c87eef075e49ba98e563e6fcc45d83 P 1dc41db601fe4b91aa6d461b504da0e7 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:14.178828  7481 tablet_replica.cc:333] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.179535  7481 raft_consensus.cc:2243] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:14.180171  7481 raft_consensus.cc:2272] T eeef0c4282c74b3692ab849adb5ce58e P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:14.181911  9297 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:14.182842  7481 tablet_replica.cc:333] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.183590  7481 raft_consensus.cc:2243] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.184788  7481 raft_consensus.cc:2272] T b9b3791a4a534a598bc36d62992d27be P 1dc41db601fe4b91aa6d461b504da0e7 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:14.187006  9297 consensus_peers.cc:597] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:14.187536  7481 tablet_replica.cc:333] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7: stopping tablet replica
I20260709 13:05:14.188325  7481 raft_consensus.cc:2243] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:14.189548  7481 raft_consensus.cc:2272] T 8ffc6f04aee94f37b24daec463e6eb45 P 1dc41db601fe4b91aa6d461b504da0e7 [term 20 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:14.292138  9296 consensus_peers.cc:597] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:14.443045  9294 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:14.587633  9296 consensus_peers.cc:597] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:14.613305  9294 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:14.736016 16767 raft_consensus.cc:493] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:14.737192 16767 raft_consensus.cc:515] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:14.740175 16767 leader_election.cc:290] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:14.750027  9294 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:14.764300  9296 consensus_peers.cc:597] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:14.767243  9296 leader_election.cc:336] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:14.768045  9296 leader_election.cc:304] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:14.769726 16767 raft_consensus.cc:2749] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:05:14.800048  9296 consensus_peers.cc:597] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:14.818619  9294 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:14.967654  9296 consensus_peers.cc:597] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:15.001458 16767 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.002573 16767 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:15.004853  9296 consensus_peers.cc:597] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:15.005903 16767 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.011381  9297 leader_election.cc:336] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
W20260709 13:05:15.021952  9296 leader_election.cc:336] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.022528  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.023414 16767 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:15.048491  9296 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:15.069612  9294 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 13:05:15.115697 16767 raft_consensus.cc:493] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.116796 16767 raft_consensus.cc:515] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.119467 16767 leader_election.cc:290] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.127708  9297 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
W20260709 13:05:15.131091  9294 leader_election.cc:336] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:15.131655  9294 leader_election.cc:304] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:05:15.132479 16767 raft_consensus.cc:2749] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:05:15.169251  9294 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 13:05:15.238756 16767 raft_consensus.cc:493] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.239724 16767 raft_consensus.cc:515] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.244475 16767 leader_election.cc:290] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.250162  9296 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.270536 16740 raft_consensus.cc:493] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.271548 16740 raft_consensus.cc:515] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.278434 16740 leader_election.cc:290] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:15.283543 16740 raft_consensus.cc:493] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.284628 16740 raft_consensus.cc:515] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.287344 16740 leader_election.cc:290] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.292330  9297 leader_election.cc:336] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.292834  9297 leader_election.cc:304] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.293661 16767 raft_consensus.cc:2749] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:15.309689 16767 raft_consensus.cc:493] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:15.312029 16767 raft_consensus.cc:515] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:15.320240  9294 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:15.327829  9296 leader_election.cc:336] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.329959  8471 maintenance_manager.cc:419] P b2a6e2bdaa2043fba12c0ef60545b7fb: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033401
W20260709 13:05:15.345010  9296 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.346846 16767 leader_election.cc:290] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:15.348450  9294 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:15.349153  9297 leader_election.cc:336] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.351600  9297 leader_election.cc:304] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:15.352828  9297 leader_election.cc:336] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.353236  9297 leader_election.cc:304] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.354688 16776 raft_consensus.cc:2749] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:15.361819 16776 raft_consensus.cc:2749] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:15.372488  9296 leader_election.cc:336] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.372999  9296 leader_election.cc:304] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.374656 16767 raft_consensus.cc:493] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.375514 16767 raft_consensus.cc:515] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.378078 16776 raft_consensus.cc:493] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.378885 16776 raft_consensus.cc:515] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:15.386543  9294 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:15.388163 16767 leader_election.cc:290] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:15.390336 16776 leader_election.cc:290] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:15.400697 16740 raft_consensus.cc:493] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.401549 16740 raft_consensus.cc:515] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.403983 16740 leader_election.cc:290] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.408747  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f4f350f118fc449fb61357528a6b14f7: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:15.408794 16740 raft_consensus.cc:2749] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:05:15.410761 16740 raft_consensus.cc:493] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.411448 16740 raft_consensus.cc:515] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:15.418466  9294 consensus_peers.cc:597] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:05:15.420253 16740 leader_election.cc:290] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.432605  9294 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:15.439575  9296 leader_election.cc:336] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:15.444597  9296 leader_election.cc:336] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:15.448028  9297 leader_election.cc:336] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.449682  9297 leader_election.cc:304] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:15.450700  9297 leader_election.cc:336] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.452241  9297 leader_election.cc:304] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:05:15.451819  9296 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Couldn't send request to peer d0a2b9c938d94a4fa258e068ea054d84. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:15.451320 16779 raft_consensus.cc:2749] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:15.455307 16740 raft_consensus.cc:2749] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:15.467175 16779 raft_consensus.cc:493] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.468113 16779 raft_consensus.cc:515] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.470558 16779 leader_election.cc:290] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.479722  9297 leader_election.cc:336] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.480271 16779 raft_consensus.cc:493] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.483723 16779 raft_consensus.cc:515] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } attrs { replace: true } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:15.490835  9294 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:15.491775  9297 leader_election.cc:304] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:15.482543  9296 leader_election.cc:336] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.493185 16740 raft_consensus.cc:2749] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:05:15.497287 16779 leader_election.cc:290] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:15.505110 16740 raft_consensus.cc:493] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.510681 16740 raft_consensus.cc:515] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.512916 16740 leader_election.cc:290] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.506718  9294 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.507171  9297 leader_election.cc:336] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.517688  9297 leader_election.cc:304] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
W20260709 13:05:15.519408  9294 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:15.520746 16740 raft_consensus.cc:2749] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:05:15.523662 16779 raft_consensus.cc:493] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:15.524915 16779 raft_consensus.cc:515] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
W20260709 13:05:15.524231  9296 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.526973 16740 raft_consensus.cc:493] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.530055 16740 raft_consensus.cc:515] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.532310 16740 leader_election.cc:290] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
I20260709 13:05:15.532896  9296 leader_election.cc:304] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.536821 16740 raft_consensus.cc:2749] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:15.543589  9294 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:15.544863  9297 leader_election.cc:336] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.546384  9297 leader_election.cc:304] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
W20260709 13:05:15.547191  9296 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.547853 16740 raft_consensus.cc:2749] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:15.549974 16779 leader_election.cc:290] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:15.562026  9297 leader_election.cc:336] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
W20260709 13:05:15.565310  9297 leader_election.cc:336] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
W20260709 13:05:15.565312  9296 leader_election.cc:336] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.565995  9297 leader_election.cc:304] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:05:15.566105  9296 leader_election.cc:304] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.562582 16779 raft_consensus.cc:493] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:15.567121 16779 raft_consensus.cc:515] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.570487  8470 maintenance_manager.cc:643] P b2a6e2bdaa2043fba12c0ef60545b7fb: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.230s	user 0.042s	sys 0.000s Metrics: {"bytes_written":6939,"delete_count":0,"lbm_write_time_us":1224,"lbm_writes_lt_1ms":6,"reinsert_count":0,"thread_start_us":77341,"threads_started":1,"update_count":285,"wal-append.queue_time_us":74104}
I20260709 13:05:15.572343 16740 raft_consensus.cc:2749] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:15.572762  8471 maintenance_manager.cc:419] P b2a6e2bdaa2043fba12c0ef60545b7fb: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.344575
W20260709 13:05:15.573313  9294 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:15.574332 16740 raft_consensus.cc:2749] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:15.570209 16779 leader_election.cc:290] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:15.590377  9297 leader_election.cc:336] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.592293  9297 leader_election.cc:304] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.593571 16779 raft_consensus.cc:2749] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:15.598079  9297 leader_election.cc:336] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
I20260709 13:05:15.598783  9297 leader_election.cc:304] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, a0d703ee18464475a290425976246555
I20260709 13:05:15.599694 16779 raft_consensus.cc:2749] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:05:15.682410  8470 maintenance_manager.cc:643] P b2a6e2bdaa2043fba12c0ef60545b7fb: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.109s	user 0.097s	sys 0.011s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":27641,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":18409,"lbm_read_time_us":1927,"lbm_reads_lt_1ms":25,"lbm_write_time_us":1490,"lbm_writes_lt_1ms":11,"peak_mem_usage":304196,"reinsert_count":0,"thread_start_us":17684,"threads_started":2,"update_count":296}
I20260709 13:05:15.684929  8471 maintenance_manager.cc:419] P b2a6e2bdaa2043fba12c0ef60545b7fb: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260709 13:05:15.685827  8470 maintenance_manager.cc:643] P b2a6e2bdaa2043fba12c0ef60545b7fb: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
I20260709 13:05:15.706677 16779 raft_consensus.cc:493] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:15.707502 16779 raft_consensus.cc:515] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.712103 16779 leader_election.cc:290] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:15.714803  9294 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:15.715291  9296 leader_election.cc:336] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:15.715703  9296 leader_election.cc:304] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.716851 16779 raft_consensus.cc:2749] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:05:15.857285 16779 raft_consensus.cc:493] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:15.858080 16779 raft_consensus.cc:515] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:15.861233 16779 leader_election.cc:290] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:15.867271  9296 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:15.867996  9294 leader_election.cc:336] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:15.868510  9294 leader_election.cc:304] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:15.869580 16740 raft_consensus.cc:2749] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:16.048817 16740 raft_consensus.cc:493] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:16.049726 16740 raft_consensus.cc:515] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:16.051975 16740 leader_election.cc:290] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:16.056851  9296 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
W20260709 13:05:16.057381  9294 leader_election.cc:336] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:16.057859  9294 leader_election.cc:304] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:16.058624 16740 raft_consensus.cc:2749] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:05:16.073784  9294 consensus_peers.cc:597] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 -> Peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Couldn't send request to peer a0d703ee18464475a290425976246555. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:16.157588  9297 consensus_peers.cc:597] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:16.182228  9297 consensus_peers.cc:597] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:16.182727  9297 consensus_peers.cc:597] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:16.227715  9297 consensus_peers.cc:597] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:16.239441  9297 consensus_peers.cc:597] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:16.260811 16740 raft_consensus.cc:493] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:16.261662 16740 raft_consensus.cc:515] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:16.264210 16740 leader_election.cc:290] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
W20260709 13:05:16.269593  9294 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:16.270421  9296 leader_election.cc:336] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:16.270913  9296 leader_election.cc:304] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:16.271682 16740 raft_consensus.cc:2749] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:05:16.288471  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
W20260709 13:05:16.366932  9297 consensus_peers.cc:597] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:16.382534  9297 consensus_peers.cc:597] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:16.388938  9297 consensus_peers.cc:597] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:16.428340  9297 consensus_peers.cc:597] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:16.448402  7481 tablet_server.cc:179] TabletServer@127.7.78.68:0 shutting down...
W20260709 13:05:16.454974  9297 consensus_peers.cc:597] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:16.499459  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:05:16.500598  7481 tablet_replica.cc:333] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.502230  7481 raft_consensus.cc:2243] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.503085  7481 raft_consensus.cc:2272] T 7778779e8695473fa6249308e7a4fa1e P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.506327  7481 tablet_replica.cc:333] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.507151  7481 raft_consensus.cc:2243] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.508352  7481 raft_consensus.cc:2272] T 121ce91dbce74f019e528db7b7062a86 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.511605  7481 tablet_replica.cc:333] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.512409  7481 raft_consensus.cc:2243] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.514394  7481 raft_consensus.cc:2272] T f4f350f118fc449fb61357528a6b14f7 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.517342  7481 tablet_replica.cc:333] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.518025  7481 raft_consensus.cc:2243] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.518867  7481 raft_consensus.cc:2272] T 081470ef13974fe497f88859ada43ffa P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.521832  7481 tablet_replica.cc:333] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.522697  7481 raft_consensus.cc:2243] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.523873  7481 raft_consensus.cc:2272] T a7c87eef075e49ba98e563e6fcc45d83 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.526610  7481 tablet_replica.cc:333] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.527279  7481 raft_consensus.cc:2243] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.528298  7481 raft_consensus.cc:2272] T ea2142382a5d4439915e07c70837c688 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.531302  7481 tablet_replica.cc:333] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.531975  7481 raft_consensus.cc:2243] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.533293  7481 raft_consensus.cc:2272] T b8b894d0fc374f33a626d5fd9d8ca4e7 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.534399 16740 raft_consensus.cc:493] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader d0a2b9c938d94a4fa258e068ea054d84)
I20260709 13:05:16.535149 16740 raft_consensus.cc:515] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a0d703ee18464475a290425976246555" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 46779 } } peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:16.537077  7481 tablet_replica.cc:333] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.537663 16740 leader_election.cc:290] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a0d703ee18464475a290425976246555 (127.7.78.65:46779), d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929)
I20260709 13:05:16.538158  7481 raft_consensus.cc:2243] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.539780  7481 raft_consensus.cc:2272] T b9b3791a4a534a598bc36d62992d27be P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:16.541885  9294 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a0d703ee18464475a290425976246555 (127.7.78.65:46779): Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:16.542981  7481 tablet_replica.cc:333] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.543939  7481 raft_consensus.cc:2243] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus shutting down.
W20260709 13:05:16.544137  9296 leader_election.cc:336] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:16.544679  9296 leader_election.cc:304] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: a0d703ee18464475a290425976246555, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:16.544952  7481 raft_consensus.cc:2272] T 04399390cf414886a3453981dc1f74aa P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.545452 16740 raft_consensus.cc:2749] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:16.548079  7481 tablet_replica.cc:333] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.549096  7481 raft_consensus.cc:2243] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.550554  7481 raft_consensus.cc:2272] T 78613f64484447c487fa71abda1ec1ac P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.553213  7481 tablet_replica.cc:333] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.553900  7481 raft_consensus.cc:2243] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 16 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.555446  7481 raft_consensus.cc:2272] T 5559db905acf4ba0b48e9e3e87f49c1c P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.558233  7481 tablet_replica.cc:333] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.559033  7481 raft_consensus.cc:2243] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.559581  7481 raft_consensus.cc:2272] T 9a389d19cf2c47d6b5873139d8cfad40 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.562173  7481 tablet_replica.cc:333] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.562778  7481 raft_consensus.cc:2243] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.563984  7481 raft_consensus.cc:2272] T 162db4080f654c9fbd8c23faf68f6df3 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.566635  7481 tablet_replica.cc:333] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.567291  7481 raft_consensus.cc:2243] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.568389  7481 raft_consensus.cc:2272] T 94c0917e28b9498a823f74b5f664a815 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.571990  7481 tablet_replica.cc:333] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.572835  7481 raft_consensus.cc:2243] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.574115  7481 raft_consensus.cc:2272] T 4b06c2e2a7c644478ef688b47c11a07a P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.576675  7481 tablet_replica.cc:333] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.577440  7481 raft_consensus.cc:2243] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.578656  7481 raft_consensus.cc:2272] T 96ad7a91a5c14870885cd58a34393f82 P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.581243  7481 tablet_replica.cc:333] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.581980  7481 raft_consensus.cc:2243] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.583281  7481 raft_consensus.cc:2272] T 4ed4dfb6b4d146c3bb0c1c3677d1b39b P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.585816  7481 tablet_replica.cc:333] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.586460  7481 raft_consensus.cc:2243] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.587250  7481 raft_consensus.cc:2272] T a346f8961f774e87a3c75fa7e2053cb0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.590951  7481 tablet_replica.cc:333] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.591933  7481 raft_consensus.cc:2243] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.592931  7481 raft_consensus.cc:2272] T b387b386886048c8b66e5f499f02aab2 P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:16.593070  9297 consensus_peers.cc:597] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 -> Peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Couldn't send request to peer 1dc41db601fe4b91aa6d461b504da0e7. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:16.595729  7481 tablet_replica.cc:333] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.596547  7481 raft_consensus.cc:2243] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.597299  7481 raft_consensus.cc:2272] T 2adcf8a687b54060b644e8c691d8772f P 453e644bad5d410ab9be85b353e28dc2 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.599840  7481 tablet_replica.cc:333] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.600449  7481 raft_consensus.cc:2243] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.601075  7481 raft_consensus.cc:2272] T eed0f0ed87f74c1fb8320d38bceeac72 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.603623  7481 tablet_replica.cc:333] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.604341  7481 raft_consensus.cc:2243] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.604976  7481 raft_consensus.cc:2272] T 9e8a9411169f407cba1b44443edc9840 P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.605237 16740 raft_consensus.cc:493] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1dc41db601fe4b91aa6d461b504da0e7)
I20260709 13:05:16.605798 16740 raft_consensus.cc:515] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0a2b9c938d94a4fa258e068ea054d84" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 39929 } } peers { permanent_uuid: "1dc41db601fe4b91aa6d461b504da0e7" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37975 } } peers { permanent_uuid: "453e644bad5d410ab9be85b353e28dc2" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 33133 } attrs { promote: false } }
I20260709 13:05:16.608526  7481 tablet_replica.cc:333] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.609694  7481 raft_consensus.cc:2243] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.610606  7481 raft_consensus.cc:2272] T a4f31f7ca2bf4416bec3531998325cc4 P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.610877 16740 leader_election.cc:290] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929), 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975)
W20260709 13:05:16.613202  9297 leader_election.cc:336] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1dc41db601fe4b91aa6d461b504da0e7 (127.7.78.67:37975): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37975: connect: Connection refused (error 111)
W20260709 13:05:16.613886  9296 leader_election.cc:336] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer d0a2b9c938d94a4fa258e068ea054d84 (127.7.78.66:39929): Network error: Client connection negotiation failed: client connection to 127.7.78.66:39929: connect: Connection refused (error 111)
I20260709 13:05:16.614279  7481 tablet_replica.cc:333] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.614367  9296 leader_election.cc:304] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 453e644bad5d410ab9be85b353e28dc2; no voters: 1dc41db601fe4b91aa6d461b504da0e7, d0a2b9c938d94a4fa258e068ea054d84
I20260709 13:05:16.615231 16740 raft_consensus.cc:2749] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:16.615437  7481 raft_consensus.cc:2243] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.616858  7481 raft_consensus.cc:2272] T c6ad38ee7735403281a14a504308a8ea P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.619220  7481 tablet_replica.cc:333] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.619899  7481 raft_consensus.cc:2243] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.620571  7481 raft_consensus.cc:2272] T 2947646ebfc34bd7a5a55312f5812ee2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.623847  7481 tablet_replica.cc:333] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.624794  7481 raft_consensus.cc:2243] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.626145  7481 raft_consensus.cc:2272] T 0337bfcbeb3b4e15b96669c8d3ca4692 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.629072  7481 tablet_replica.cc:333] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.629920  7481 raft_consensus.cc:2243] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.631317  7481 raft_consensus.cc:2272] T c3d38a98ca324ddbbb3ff25fb6ded965 P 453e644bad5d410ab9be85b353e28dc2 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.633848  7481 tablet_replica.cc:333] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.634481  7481 raft_consensus.cc:2243] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.635691  7481 raft_consensus.cc:2272] T 689ad8326d634dee8402f1352a76f0e2 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.638248  7481 tablet_replica.cc:333] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.638939  7481 raft_consensus.cc:2243] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.639804  7481 raft_consensus.cc:2272] T 03f3a12327224113b1c5bf090e280a53 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.642349  7481 tablet_replica.cc:333] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.642982  7481 raft_consensus.cc:2243] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.643805  7481 raft_consensus.cc:2272] T 266236cfcd824409b757505321b36d81 P 453e644bad5d410ab9be85b353e28dc2 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.646063  7481 tablet_replica.cc:333] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.646701  7481 raft_consensus.cc:2243] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.647331  7481 raft_consensus.cc:2272] T bf63362bb9b4409299528319b7472b04 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.649752  7481 tablet_replica.cc:333] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.650352  7481 raft_consensus.cc:2243] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.650914  7481 raft_consensus.cc:2272] T cde36bf9fcc5409e8a00b747fbfbfb06 P 453e644bad5d410ab9be85b353e28dc2 [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.653765  7481 tablet_replica.cc:333] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.654649  7481 raft_consensus.cc:2243] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.655335  7481 raft_consensus.cc:2272] T d35f944458a44f6c82498c7fec026ec5 P 453e644bad5d410ab9be85b353e28dc2 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.657859  7481 tablet_replica.cc:333] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.658878  7481 raft_consensus.cc:2243] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.660152  7481 raft_consensus.cc:2272] T 98c31f9999e84270898dfb87c21cbc7c P 453e644bad5d410ab9be85b353e28dc2 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.662532  7481 tablet_replica.cc:333] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.663149  7481 raft_consensus.cc:2243] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.664253  7481 raft_consensus.cc:2272] T 77d6200607074a5ebd3e4ae470a2d6ed P 453e644bad5d410ab9be85b353e28dc2 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.666931  7481 tablet_replica.cc:333] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.667549  7481 raft_consensus.cc:2243] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.668185  7481 raft_consensus.cc:2272] T dd0fc940d95d4b118d3e2564ed48b09e P 453e644bad5d410ab9be85b353e28dc2 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.671384  7481 tablet_replica.cc:333] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.672041  7481 raft_consensus.cc:2243] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.672598  7481 raft_consensus.cc:2272] T 9f9dd5b88e0647f284658784381e1893 P 453e644bad5d410ab9be85b353e28dc2 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.675012  7481 tablet_replica.cc:333] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.675570  7481 raft_consensus.cc:2243] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:16.676759  7481 raft_consensus.cc:2272] T 3ad2aa6046e746348b02a2571b7580f0 P 453e644bad5d410ab9be85b353e28dc2 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:16.679291  7481 tablet_replica.cc:333] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2: stopping tablet replica
I20260709 13:05:16.679827  7481 raft_consensus.cc:2243] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:16.680419  7481 raft_consensus.cc:2272] T 4d18196b7e714cb6a79ec64ea3d65803 P 453e644bad5d410ab9be85b353e28dc2 [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:17.208189  7481 tablet_server.cc:196] TabletServer@127.7.78.68:0 shutdown complete.
I20260709 13:05:17.283327  7481 master.cc:562] Master@127.7.78.126:44983 shutting down...
W20260709 13:05:17.330598  8530 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 13:05:17.330933  8530 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:05:17.331115  8530 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:05:29.656669  8466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111) [suppressed 247 similar messages]
W20260709 13:05:29.664657  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 121ce91dbce74f019e528db7b7062a86: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
W20260709 13:05:31.828073  8466 catalog_manager.cc:4755] TS a0d703ee18464475a290425976246555 (127.7.78.65:46779): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f4f350f118fc449fb61357528a6b14f7: Network error: Client connection negotiation failed: client connection to 127.7.78.65:46779: connect: Connection refused (error 111)
I20260709 13:05:34.343973  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:34.345335  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:34.345824  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P b2a6e2bdaa2043fba12c0ef60545b7fb: stopping tablet replica
I20260709 13:05:34.635129  7481 master.cc:584] Master@127.7.78.126:44983 shutdown complete.
I20260709 13:05:34.702037  7481 test_util.cc:182] -----------------------------------------------
I20260709 13:05:34.702273  7481 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602031409637-7481-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (279658 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 13:05:34.706075  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:37799
I20260709 13:05:34.707331  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:05:34.714591 16789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:34.714583 16790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:34.714913 16792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:34.716264  7481 server_base.cc:1061] running on GCE node
I20260709 13:05:34.719977  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:05:34.720227  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:05:34.720389  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602334720371 us; error 0 us; skew 500 ppm
I20260709 13:05:34.720917  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:05:34.723774  7481 webserver.cc:533] Webserver started at http://127.7.78.126:35775/ using document root <none> and password file <none>
I20260709 13:05:34.724306  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:05:34.724512  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:05:34.724808  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:05:34.726183  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "cd12eb484b26430aa64e7db64cb43f07"
format_stamp: "Formatted at 2026-07-09 13:05:34 on dist-test-slave-50cs"
I20260709 13:05:34.731087  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:05:34.735481 16797 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:34.736639  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:05:34.737140  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "cd12eb484b26430aa64e7db64cb43f07"
format_stamp: "Formatted at 2026-07-09 13:05:34 on dist-test-slave-50cs"
I20260709 13:05:34.737524  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:05:34.749727  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:05:34.751013  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:05:34.799717  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:37799
I20260709 13:05:34.799862 16848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:37799 every 8 connection(s)
I20260709 13:05:34.804831 16849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:34.820021 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07: Bootstrap starting.
I20260709 13:05:34.825875 16849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:34.831314 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07: No bootstrap required, opened a new log
I20260709 13:05:34.834132 16849 raft_consensus.cc:359] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd12eb484b26430aa64e7db64cb43f07" member_type: VOTER }
I20260709 13:05:34.834707 16849 raft_consensus.cc:385] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:34.834909 16849 raft_consensus.cc:740] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd12eb484b26430aa64e7db64cb43f07, State: Initialized, Role: FOLLOWER
I20260709 13:05:34.835565 16849 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd12eb484b26430aa64e7db64cb43f07" member_type: VOTER }
I20260709 13:05:34.836071 16849 raft_consensus.cc:399] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:05:34.836297 16849 raft_consensus.cc:493] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:05:34.836530 16849 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:34.842474 16849 raft_consensus.cc:515] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd12eb484b26430aa64e7db64cb43f07" member_type: VOTER }
I20260709 13:05:34.843187 16849 leader_election.cc:304] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd12eb484b26430aa64e7db64cb43f07; no voters: 
I20260709 13:05:34.844558 16849 leader_election.cc:290] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:05:34.845000 16852 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:34.846678 16852 raft_consensus.cc:697] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 1 LEADER]: Becoming Leader. State: Replica: cd12eb484b26430aa64e7db64cb43f07, State: Running, Role: LEADER
I20260709 13:05:34.847478 16852 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd12eb484b26430aa64e7db64cb43f07" member_type: VOTER }
I20260709 13:05:34.848018 16849 sys_catalog.cc:565] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:05:34.854715 16853 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd12eb484b26430aa64e7db64cb43f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd12eb484b26430aa64e7db64cb43f07" member_type: VOTER } }
I20260709 13:05:34.855480 16853 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [sys.catalog]: This master's current role is: LEADER
I20260709 13:05:34.856165 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd12eb484b26430aa64e7db64cb43f07. Latest consensus state: current_term: 1 leader_uuid: "cd12eb484b26430aa64e7db64cb43f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd12eb484b26430aa64e7db64cb43f07" member_type: VOTER } }
I20260709 13:05:34.856845 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [sys.catalog]: This master's current role is: LEADER
I20260709 13:05:34.858875 16860 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:05:34.867707 16860 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:05:34.868973  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:05:34.879112 16860 catalog_manager.cc:1383] Generated new cluster ID: 1be18bf1e6f14732b3a5f69c15498cdd
I20260709 13:05:34.879534 16860 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:05:34.894740 16860 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:05:34.896414 16860 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:05:34.911465 16860 catalog_manager.cc:6070] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07: Generated new TSK 0
I20260709 13:05:34.912309 16860 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:05:34.935916  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:05:34.943401 16870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:34.945139 16871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:34.947077 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:34.948163  7481 server_base.cc:1061] running on GCE node
I20260709 13:05:34.951697  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:05:34.952006  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:05:34.952186  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602334952169 us; error 0 us; skew 500 ppm
I20260709 13:05:34.952786  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:05:34.955907  7481 webserver.cc:533] Webserver started at http://127.7.78.65:33595/ using document root <none> and password file <none>
I20260709 13:05:34.956539  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:05:34.956768  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:05:34.957091  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:05:34.958839  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "71a10d4adec64d58ba645d68425ed531"
format_stamp: "Formatted at 2026-07-09 13:05:34 on dist-test-slave-50cs"
I20260709 13:05:34.964422  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 13:05:34.968829 16878 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:34.969990  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:05:34.970440  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "71a10d4adec64d58ba645d68425ed531"
format_stamp: "Formatted at 2026-07-09 13:05:34 on dist-test-slave-50cs"
I20260709 13:05:34.970880  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:05:34.996286  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:05:34.997871  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:05:34.999604  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:05:35.002646  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:05:35.002887  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.003248  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:05:35.003409  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.052701  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:36803
I20260709 13:05:35.052845 16940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:36803 every 8 connection(s)
I20260709 13:05:35.058210  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:05:35.066609 16945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:35.067945 16946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:35.072914  7481 server_base.cc:1061] running on GCE node
W20260709 13:05:35.073817 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:35.076120  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:05:35.076431  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:05:35.076642  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602335076620 us; error 0 us; skew 500 ppm
I20260709 13:05:35.077471  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:05:35.079842 16941 heartbeater.cc:344] Connected to a master server at 127.7.78.126:37799
I20260709 13:05:35.080384 16941 heartbeater.cc:461] Registering TS with master...
I20260709 13:05:35.080715  7481 webserver.cc:533] Webserver started at http://127.7.78.66:44419/ using document root <none> and password file <none>
I20260709 13:05:35.081275  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:05:35.081506  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:05:35.081346 16941 heartbeater.cc:507] Master 127.7.78.126:37799 requested a full tablet report, sending...
I20260709 13:05:35.081959  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:05:35.083366  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "c9521ca6efa94c3e9f074df69170cd5b"
format_stamp: "Formatted at 2026-07-09 13:05:35 on dist-test-slave-50cs"
I20260709 13:05:35.084282 16814 ts_manager.cc:194] Registered new tserver with Master: 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:35.087025 16814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58366
I20260709 13:05:35.090155  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.005s
I20260709 13:05:35.094573 16953 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.095711  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:05:35.096087  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "c9521ca6efa94c3e9f074df69170cd5b"
format_stamp: "Formatted at 2026-07-09 13:05:35 on dist-test-slave-50cs"
I20260709 13:05:35.096417  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:05:35.108366  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:05:35.109750  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:05:35.111326  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:05:35.113791  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:05:35.114024  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.114297  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:05:35.114472  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.159087  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:41991
I20260709 13:05:35.159202 17015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:41991 every 8 connection(s)
I20260709 13:05:35.164484  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:05:35.174401 17020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:35.176317 17019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:35.180040 17016 heartbeater.cc:344] Connected to a master server at 127.7.78.126:37799
I20260709 13:05:35.180619 17016 heartbeater.cc:461] Registering TS with master...
I20260709 13:05:35.180883  7481 server_base.cc:1061] running on GCE node
I20260709 13:05:35.181766 17016 heartbeater.cc:507] Master 127.7.78.126:37799 requested a full tablet report, sending...
W20260709 13:05:35.182758 17022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:35.184188  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:05:35.184468  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:05:35.184599 16814 ts_manager.cc:194] Registered new tserver with Master: c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:35.184684  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602335184664 us; error 0 us; skew 500 ppm
I20260709 13:05:35.185693  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:05:35.186547 16814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58382
I20260709 13:05:35.188612  7481 webserver.cc:533] Webserver started at http://127.7.78.67:43671/ using document root <none> and password file <none>
I20260709 13:05:35.189318  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:05:35.189585  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:05:35.189890  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:05:35.191046  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
format_stamp: "Formatted at 2026-07-09 13:05:35 on dist-test-slave-50cs"
I20260709 13:05:35.195762  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:05:35.199731 17027 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.200735  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:05:35.201113  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
format_stamp: "Formatted at 2026-07-09 13:05:35 on dist-test-slave-50cs"
I20260709 13:05:35.201493  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:05:35.215102  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:05:35.216465  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:05:35.218392  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:05:35.220908  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:05:35.221190  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.221530  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:05:35.221732  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.268437  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:37727
I20260709 13:05:35.268587 17089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:37727 every 8 connection(s)
I20260709 13:05:35.273890  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:05:35.283849 17094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:35.283890 17093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:35.287475 17096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:35.288165  7481 server_base.cc:1061] running on GCE node
I20260709 13:05:35.288823 17090 heartbeater.cc:344] Connected to a master server at 127.7.78.126:37799
I20260709 13:05:35.289218  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:05:35.289283 17090 heartbeater.cc:461] Registering TS with master...
W20260709 13:05:35.289718  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:05:35.289954  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602335289933 us; error 0 us; skew 500 ppm
I20260709 13:05:35.290506 17090 heartbeater.cc:507] Master 127.7.78.126:37799 requested a full tablet report, sending...
I20260709 13:05:35.290843  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:05:35.293383 16814 ts_manager.cc:194] Registered new tserver with Master: f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:35.294070  7481 webserver.cc:533] Webserver started at http://127.7.78.68:43775/ using document root <none> and password file <none>
I20260709 13:05:35.294641  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:05:35.294855  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:05:35.295126  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:05:35.295291 16814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58384
I20260709 13:05:35.296697  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-3-root/instance:
uuid: "40c8156681dd4a7bb2d037afbeb01580"
format_stamp: "Formatted at 2026-07-09 13:05:35 on dist-test-slave-50cs"
I20260709 13:05:35.302030  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.000s
I20260709 13:05:35.305867 17101 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.306802  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 13:05:35.307231  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-3-root
uuid: "40c8156681dd4a7bb2d037afbeb01580"
format_stamp: "Formatted at 2026-07-09 13:05:35 on dist-test-slave-50cs"
I20260709 13:05:35.307546  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:05:35.324462  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:05:35.325927  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:05:35.327445  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:05:35.329981  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:05:35.330263  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.330521  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:05:35.330696  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:35.381578  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.68:39323
I20260709 13:05:35.381707 17163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.68:39323 every 8 connection(s)
I20260709 13:05:35.396536 17164 heartbeater.cc:344] Connected to a master server at 127.7.78.126:37799
I20260709 13:05:35.397064 17164 heartbeater.cc:461] Registering TS with master...
I20260709 13:05:35.398038 17164 heartbeater.cc:507] Master 127.7.78.126:37799 requested a full tablet report, sending...
I20260709 13:05:35.400141 16814 ts_manager.cc:194] Registered new tserver with Master: 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:35.400521  7481 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.015403355s
I20260709 13:05:35.402005 16814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58396
I20260709 13:05:35.431978 16814 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58398:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:05:35.863190 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:35.895092 16906 tablet_service.cc:1511] Processing CreateTablet for tablet d582455c79c94516b94ee97bd42c5078 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:05:35.896365 16906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d582455c79c94516b94ee97bd42c5078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.903398 16905 tablet_service.cc:1511] Processing CreateTablet for tablet cd439fd5f4d4479f9d2c2003abed1c4c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:05:35.905156 16905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd439fd5f4d4479f9d2c2003abed1c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.915596 16904 tablet_service.cc:1511] Processing CreateTablet for tablet 277a6b1c93854935be35e234b98330d2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:05:35.917066 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 277a6b1c93854935be35e234b98330d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.924250 16903 tablet_service.cc:1511] Processing CreateTablet for tablet 46c332c6b3e148348daba6e06b3bc967 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:05:35.925653 16903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c332c6b3e148348daba6e06b3bc967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.926025 16980 tablet_service.cc:1511] Processing CreateTablet for tablet cd439fd5f4d4479f9d2c2003abed1c4c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:05:35.927534 16980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd439fd5f4d4479f9d2c2003abed1c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.930598 16981 tablet_service.cc:1511] Processing CreateTablet for tablet d582455c79c94516b94ee97bd42c5078 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:05:35.931042 16979 tablet_service.cc:1511] Processing CreateTablet for tablet 2521b7f56acb452087e4eccf58dd0f4a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:05:35.931988 16981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d582455c79c94516b94ee97bd42c5078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.933094 16979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2521b7f56acb452087e4eccf58dd0f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.936024 16902 tablet_service.cc:1511] Processing CreateTablet for tablet 0f90b95672134fac819741b525d2931b (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:05:35.938570 16901 tablet_service.cc:1511] Processing CreateTablet for tablet 6c0204bc7fd34bcda060984d39b24e99 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:05:35.939239 17129 tablet_service.cc:1511] Processing CreateTablet for tablet cd439fd5f4d4479f9d2c2003abed1c4c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:05:35.940766 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd439fd5f4d4479f9d2c2003abed1c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.940871 17125 tablet_service.cc:1511] Processing CreateTablet for tablet 6c0204bc7fd34bcda060984d39b24e99 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:05:35.942303 17125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0204bc7fd34bcda060984d39b24e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.942605 17128 tablet_service.cc:1511] Processing CreateTablet for tablet 277a6b1c93854935be35e234b98330d2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:05:35.943969 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 277a6b1c93854935be35e234b98330d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.946532 16978 tablet_service.cc:1511] Processing CreateTablet for tablet 46c332c6b3e148348daba6e06b3bc967 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:05:35.948014 16978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c332c6b3e148348daba6e06b3bc967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.947724 17126 tablet_service.cc:1511] Processing CreateTablet for tablet 0f90b95672134fac819741b525d2931b (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:05:35.948376 16900 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3da49eeaa7457e8f3b1848d46bcea5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:05:35.949100 17126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f90b95672134fac819741b525d2931b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.946535 17127 tablet_service.cc:1511] Processing CreateTablet for tablet 2521b7f56acb452087e4eccf58dd0f4a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:05:35.950619 17127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2521b7f56acb452087e4eccf58dd0f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.950817 16899 tablet_service.cc:1511] Processing CreateTablet for tablet 6062bb4013504733ac85a70a2ae0ff32 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:05:35.952396 16902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f90b95672134fac819741b525d2931b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.955435 17124 tablet_service.cc:1511] Processing CreateTablet for tablet 2c4f7877dcc446269524421e26325689 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:05:35.956956 17124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4f7877dcc446269524421e26325689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.957885 16977 tablet_service.cc:1511] Processing CreateTablet for tablet 0f90b95672134fac819741b525d2931b (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:05:35.958451 16976 tablet_service.cc:1511] Processing CreateTablet for tablet 2c4f7877dcc446269524421e26325689 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:05:35.958621 17123 tablet_service.cc:1511] Processing CreateTablet for tablet 6062bb4013504733ac85a70a2ae0ff32 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:05:35.959358 16977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f90b95672134fac819741b525d2931b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.960111 17123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6062bb4013504733ac85a70a2ae0ff32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.960803 16975 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3da49eeaa7457e8f3b1848d46bcea5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:05:35.961311 16974 tablet_service.cc:1511] Processing CreateTablet for tablet 6062bb4013504733ac85a70a2ae0ff32 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:05:35.962277 16975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3da49eeaa7457e8f3b1848d46bcea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.962975 16974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6062bb4013504733ac85a70a2ae0ff32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.963135 17122 tablet_service.cc:1511] Processing CreateTablet for tablet 5afc790981c04080acad7b7cd96544eb (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:05:35.964154 16899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6062bb4013504733ac85a70a2ae0ff32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.964591 17122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afc790981c04080acad7b7cd96544eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.965811 16973 tablet_service.cc:1511] Processing CreateTablet for tablet 9915d72aefda43ec9b9f65e31cf0b9ea (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:05:35.969820 16901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0204bc7fd34bcda060984d39b24e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.971549 16972 tablet_service.cc:1511] Processing CreateTablet for tablet 705e2e01faca442ea62979654d514c91 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:05:35.975266 17121 tablet_service.cc:1511] Processing CreateTablet for tablet 9915d72aefda43ec9b9f65e31cf0b9ea (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:05:35.976845 17121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9915d72aefda43ec9b9f65e31cf0b9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.963652 16976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4f7877dcc446269524421e26325689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.977725 16900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3da49eeaa7457e8f3b1848d46bcea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.981835 16972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705e2e01faca442ea62979654d514c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.983408 16973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9915d72aefda43ec9b9f65e31cf0b9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:35.983880 16898 tablet_service.cc:1511] Processing CreateTablet for tablet 5afc790981c04080acad7b7cd96544eb (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:05:35.985639 16898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afc790981c04080acad7b7cd96544eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.003733 16897 tablet_service.cc:1511] Processing CreateTablet for tablet 705e2e01faca442ea62979654d514c91 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:05:36.005939 16897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705e2e01faca442ea62979654d514c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.013890 17120 tablet_service.cc:1511] Processing CreateTablet for tablet c91680aaf7f642fb8b9697eaa08d67ac (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:05:36.015472 17120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91680aaf7f642fb8b9697eaa08d67ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.090768 16941 heartbeater.cc:499] Master 127.7.78.126:37799 was elected leader, sending a full tablet report...
I20260709 13:05:36.105473 17055 tablet_service.cc:1511] Processing CreateTablet for tablet d582455c79c94516b94ee97bd42c5078 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:05:36.106102 17054 tablet_service.cc:1511] Processing CreateTablet for tablet 277a6b1c93854935be35e234b98330d2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:05:36.107182 17055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d582455c79c94516b94ee97bd42c5078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.108120 17052 tablet_service.cc:1511] Processing CreateTablet for tablet 46c332c6b3e148348daba6e06b3bc967 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:05:36.108711 17053 tablet_service.cc:1511] Processing CreateTablet for tablet 2521b7f56acb452087e4eccf58dd0f4a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:05:36.110647 17054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 277a6b1c93854935be35e234b98330d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.112044 17049 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3da49eeaa7457e8f3b1848d46bcea5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:05:36.110575 17050 tablet_service.cc:1511] Processing CreateTablet for tablet 2c4f7877dcc446269524421e26325689 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:05:36.113658 17049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3da49eeaa7457e8f3b1848d46bcea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.114394 17050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4f7877dcc446269524421e26325689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.110671 17051 tablet_service.cc:1511] Processing CreateTablet for tablet 6c0204bc7fd34bcda060984d39b24e99 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:05:36.116714 17048 tablet_service.cc:1511] Processing CreateTablet for tablet 5afc790981c04080acad7b7cd96544eb (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:05:36.117813 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0204bc7fd34bcda060984d39b24e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.118939 17048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afc790981c04080acad7b7cd96544eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.122133 17053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2521b7f56acb452087e4eccf58dd0f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.122255 17124 tablet_service.cc:1511] Processing CreateTablet for tablet 65c685d378244491a760363b4eec896e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:05:36.122278 16977 tablet_service.cc:1511] Processing CreateTablet for tablet 3e32f5ae45c549ef88703600d6a9cde3 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:05:36.124009 17124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c685d378244491a760363b4eec896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.124027 16977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e32f5ae45c549ef88703600d6a9cde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.125780 17052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c332c6b3e148348daba6e06b3bc967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.133867 16899 tablet_service.cc:1511] Processing CreateTablet for tablet 65c685d378244491a760363b4eec896e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:05:36.134799 17047 tablet_service.cc:1511] Processing CreateTablet for tablet 9915d72aefda43ec9b9f65e31cf0b9ea (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:05:36.136284 17047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9915d72aefda43ec9b9f65e31cf0b9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.135495 16899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c685d378244491a760363b4eec896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.139454 17046 tablet_service.cc:1511] Processing CreateTablet for tablet 705e2e01faca442ea62979654d514c91 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:05:36.140967 17046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705e2e01faca442ea62979654d514c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.152304 16904 tablet_service.cc:1511] Processing CreateTablet for tablet 4d9193bb7f35422380ce6a3fb390eef9 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:05:36.152751 16900 tablet_service.cc:1511] Processing CreateTablet for tablet c91680aaf7f642fb8b9697eaa08d67ac (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:05:36.154738 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9193bb7f35422380ce6a3fb390eef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.154525 17126 tablet_service.cc:1511] Processing CreateTablet for tablet 7e5eff39a6814f6e8a37a48064b3dd2e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:05:36.156044 17126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5eff39a6814f6e8a37a48064b3dd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.158051 16900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91680aaf7f642fb8b9697eaa08d67ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.201520 17016 heartbeater.cc:499] Master 127.7.78.126:37799 was elected leader, sending a full tablet report...
W20260709 13:05:36.208506 16801 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 47963.1us
I20260709 13:05:36.210166 17185 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:36.217162 17185 tablet_bootstrap.cc:654] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:36.222478 16980 tablet_service.cc:1511] Processing CreateTablet for tablet 4d9193bb7f35422380ce6a3fb390eef9 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:05:36.226293 16980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9193bb7f35422380ce6a3fb390eef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.223362 16979 tablet_service.cc:1511] Processing CreateTablet for tablet c91680aaf7f642fb8b9697eaa08d67ac (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:05:36.230248 16979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91680aaf7f642fb8b9697eaa08d67ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.231896 17184 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:36.243974 16902 tablet_service.cc:1511] Processing CreateTablet for tablet 7e5eff39a6814f6e8a37a48064b3dd2e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:05:36.249987 16902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5eff39a6814f6e8a37a48064b3dd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.258442 16904 tablet_service.cc:1511] Processing CreateTablet for tablet 5db30b4886e54685a69ed279bec9df94 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:05:36.259949 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db30b4886e54685a69ed279bec9df94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.258898 16973 tablet_service.cc:1511] Processing CreateTablet for tablet 7e5eff39a6814f6e8a37a48064b3dd2e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:05:36.262866 16973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5eff39a6814f6e8a37a48064b3dd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.141052 17121 tablet_service.cc:1511] Processing CreateTablet for tablet 3e32f5ae45c549ef88703600d6a9cde3 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:05:36.265735 17048 tablet_service.cc:1511] Processing CreateTablet for tablet 4d9193bb7f35422380ce6a3fb390eef9 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:05:36.268739 16972 tablet_service.cc:1511] Processing CreateTablet for tablet c38d4f3da4f646a285221f607421196a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:05:36.273731 17121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e32f5ae45c549ef88703600d6a9cde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.277908 16972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38d4f3da4f646a285221f607421196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.282223 17048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9193bb7f35422380ce6a3fb390eef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.286265 17183 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
W20260709 13:05:36.289559 16798 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 45567us
I20260709 13:05:36.298399 17090 heartbeater.cc:499] Master 127.7.78.126:37799 was elected leader, sending a full tablet report...
I20260709 13:05:36.303072 17127 tablet_service.cc:1511] Processing CreateTablet for tablet 5db30b4886e54685a69ed279bec9df94 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:05:36.306778 17127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db30b4886e54685a69ed279bec9df94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.315105 17186 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:36.316121 17051 tablet_service.cc:1511] Processing CreateTablet for tablet 65c685d378244491a760363b4eec896e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:05:36.317531 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65c685d378244491a760363b4eec896e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.325850 17128 tablet_service.cc:1511] Processing CreateTablet for tablet c38d4f3da4f646a285221f607421196a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:05:36.327342 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38d4f3da4f646a285221f607421196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.330440 16905 tablet_service.cc:1511] Processing CreateTablet for tablet 95585e8984cc45859857b2ff7777c409 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:05:36.331837 16905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95585e8984cc45859857b2ff7777c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.336798 17183 tablet_bootstrap.cc:654] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:36.352216 17184 tablet_bootstrap.cc:654] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:36.363926 17049 tablet_service.cc:1511] Processing CreateTablet for tablet 3e32f5ae45c549ef88703600d6a9cde3 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:05:36.365989 17049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e32f5ae45c549ef88703600d6a9cde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.369292 16977 tablet_service.cc:1511] Processing CreateTablet for tablet 95585e8984cc45859857b2ff7777c409 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:05:36.384312 17186 tablet_bootstrap.cc:654] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:36.385228 16977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95585e8984cc45859857b2ff7777c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.386643 17052 tablet_service.cc:1511] Processing CreateTablet for tablet 5db30b4886e54685a69ed279bec9df94 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:05:36.388103 17052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db30b4886e54685a69ed279bec9df94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.392474 17055 tablet_service.cc:1511] Processing CreateTablet for tablet c38d4f3da4f646a285221f607421196a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:05:36.405501 17164 heartbeater.cc:499] Master 127.7.78.126:37799 was elected leader, sending a full tablet report...
I20260709 13:05:36.422930 17185 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:36.439767 16899 tablet_service.cc:1511] Processing CreateTablet for tablet a3fbc4f00185436cbce1d8e24e6d1e3a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:05:36.443432 17047 tablet_service.cc:1511] Processing CreateTablet for tablet 95585e8984cc45859857b2ff7777c409 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:05:36.442274 17184 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:36.446102 17184 ts_tablet_manager.cc:1403] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.240s	user 0.014s	sys 0.028s
I20260709 13:05:36.453110 16899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fbc4f00185436cbce1d8e24e6d1e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.434448 17055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38d4f3da4f646a285221f607421196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.458089 17047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95585e8984cc45859857b2ff7777c409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.461745 17128 tablet_service.cc:1511] Processing CreateTablet for tablet a3fbc4f00185436cbce1d8e24e6d1e3a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:05:36.463241 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fbc4f00185436cbce1d8e24e6d1e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.463397 17183 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:36.464053 17183 ts_tablet_manager.cc:1403] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.178s	user 0.014s	sys 0.016s
I20260709 13:05:36.467638 17183 raft_consensus.cc:359] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.475137 17183 raft_consensus.cc:385] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:36.475365 16979 tablet_service.cc:1511] Processing CreateTablet for tablet a3fbc4f00185436cbce1d8e24e6d1e3a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:05:36.475688 17183 raft_consensus.cc:740] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:36.459199 17185 ts_tablet_manager.cc:1403] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.249s	user 0.016s	sys 0.005s
I20260709 13:05:36.474004 17054 tablet_service.cc:1511] Processing CreateTablet for tablet 71be301ca5434f9b8880e7a03ef1ebca (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:05:36.477957 17054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71be301ca5434f9b8880e7a03ef1ebca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.452422 17184 raft_consensus.cc:359] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:36.478860 17184 raft_consensus.cc:385] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:36.479266 17184 raft_consensus.cc:740] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:36.476716 16979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fbc4f00185436cbce1d8e24e6d1e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.480067 17185 raft_consensus.cc:359] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.489948 17185 raft_consensus.cc:385] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:36.471908 17124 tablet_service.cc:1511] Processing CreateTablet for tablet 71be301ca5434f9b8880e7a03ef1ebca (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:05:36.477619 17183 consensus_queue.cc:260] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.492527 16973 tablet_service.cc:1511] Processing CreateTablet for tablet f1adb90cc0554b10b2f20940047d6fc2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:05:36.458153 16900 tablet_service.cc:1511] Processing CreateTablet for tablet 71be301ca5434f9b8880e7a03ef1ebca (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:05:36.511543 16973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1adb90cc0554b10b2f20940047d6fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.512529 16900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71be301ca5434f9b8880e7a03ef1ebca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.493813 17046 tablet_service.cc:1511] Processing CreateTablet for tablet f1adb90cc0554b10b2f20940047d6fc2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:05:36.514415 17046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1adb90cc0554b10b2f20940047d6fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.515219 17124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71be301ca5434f9b8880e7a03ef1ebca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.517017 17121 tablet_service.cc:1511] Processing CreateTablet for tablet b2784691feee4fdc945ee8ea311ea7da (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:05:36.480149 17184 consensus_queue.cc:260] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:36.498060 17185 raft_consensus.cc:740] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:36.518410 17121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2784691feee4fdc945ee8ea311ea7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.501096 17186 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:36.519469 17185 consensus_queue.cc:260] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.520242 17186 ts_tablet_manager.cc:1403] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.206s	user 0.017s	sys 0.000s
I20260709 13:05:36.524238 17186 raft_consensus.cc:359] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.525153 17186 raft_consensus.cc:385] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:36.496376 17126 tablet_service.cc:1511] Processing CreateTablet for tablet f1adb90cc0554b10b2f20940047d6fc2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:05:36.496376 16904 tablet_service.cc:1511] Processing CreateTablet for tablet 2c6703b86b3e4b4fad39ae2660fe6ff8 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:05:36.532100 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6703b86b3e4b4fad39ae2660fe6ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.532110 17126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1adb90cc0554b10b2f20940047d6fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.546468 16977 tablet_service.cc:1511] Processing CreateTablet for tablet 2c6703b86b3e4b4fad39ae2660fe6ff8 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:05:36.550457 17052 tablet_service.cc:1511] Processing CreateTablet for tablet 2c6703b86b3e4b4fad39ae2660fe6ff8 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:05:36.578433 17052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6703b86b3e4b4fad39ae2660fe6ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.578763 16975 tablet_service.cc:1511] Processing CreateTablet for tablet f190fb92000c417a92080b56288b075f (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:05:36.580025 16975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f190fb92000c417a92080b56288b075f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.580830 17049 tablet_service.cc:1511] Processing CreateTablet for tablet f190fb92000c417a92080b56288b075f (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:05:36.582273 17049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f190fb92000c417a92080b56288b075f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.558347 16902 tablet_service.cc:1511] Processing CreateTablet for tablet ffcadf1be2784b748c215778456c2384 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:05:36.558506 16905 tablet_service.cc:1511] Processing CreateTablet for tablet b2784691feee4fdc945ee8ea311ea7da (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:05:36.529644 17186 raft_consensus.cc:740] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:36.541889 17185 ts_tablet_manager.cc:1434] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.065s	user 0.013s	sys 0.001s
I20260709 13:05:36.586450 17185 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:36.559669 16972 tablet_service.cc:1511] Processing CreateTablet for tablet b2784691feee4fdc945ee8ea311ea7da (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:05:36.587524 16972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2784691feee4fdc945ee8ea311ea7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.560509 16980 tablet_service.cc:1511] Processing CreateTablet for tablet 3f4a7e79870e405e9938b891f964fd1d (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:05:36.591843 16980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4a7e79870e405e9938b891f964fd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.561534 17128 tablet_service.cc:1511] Processing CreateTablet for tablet ffcadf1be2784b748c215778456c2384 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:05:36.594663 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffcadf1be2784b748c215778456c2384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.565315 17054 tablet_service.cc:1511] Processing CreateTablet for tablet 3f4a7e79870e405e9938b891f964fd1d (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:05:36.598196 17054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4a7e79870e405e9938b891f964fd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.566413 17055 tablet_service.cc:1511] Processing CreateTablet for tablet ffcadf1be2784b748c215778456c2384 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:05:36.567767 17127 tablet_service.cc:1511] Processing CreateTablet for tablet 3f4a7e79870e405e9938b891f964fd1d (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:05:36.569901 17183 ts_tablet_manager.cc:1434] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.105s	user 0.018s	sys 0.004s
I20260709 13:05:36.574541 16899 tablet_service.cc:1511] Processing CreateTablet for tablet f190fb92000c417a92080b56288b075f (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:05:36.604293 17055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffcadf1be2784b748c215778456c2384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.604077 16904 tablet_service.cc:1511] Processing CreateTablet for tablet af54823c7c644c92b460fde468cdfaa0 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:05:36.607033 17184 ts_tablet_manager.cc:1434] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.157s	user 0.006s	sys 0.034s
W20260709 13:05:36.607527 16942 tablet.cc:2406] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:05:36.585156 16902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffcadf1be2784b748c215778456c2384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.601872 17127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4a7e79870e405e9938b891f964fd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.602144 17125 tablet_service.cc:1511] Processing CreateTablet for tablet af54823c7c644c92b460fde468cdfaa0 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:05:36.614141 16979 tablet_service.cc:1511] Processing CreateTablet for tablet af54823c7c644c92b460fde468cdfaa0 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:05:36.615036 17125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af54823c7c644c92b460fde468cdfaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.602944 17183 tablet_bootstrap.cc:492] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:36.618218 17191 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:36.618819 17191 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.621613 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af54823c7c644c92b460fde468cdfaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.625802 16899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f190fb92000c417a92080b56288b075f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.638837 17186 consensus_queue.cc:260] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.640812 16905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2784691feee4fdc945ee8ea311ea7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.631733 17191 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:36.654212 16977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6703b86b3e4b4fad39ae2660fe6ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.670568 16897 tablet_service.cc:1511] Processing CreateTablet for tablet 61b68459f0de48d98ad40888859c37f5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:05:36.672155 16897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b68459f0de48d98ad40888859c37f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.681802 17184 tablet_bootstrap.cc:492] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:36.690150 17052 tablet_service.cc:1511] Processing CreateTablet for tablet 61b68459f0de48d98ad40888859c37f5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:05:36.702934 16978 tablet_service.cc:1511] Processing CreateTablet for tablet f4cf831311924bb7b8afbc30043e1550 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:05:36.704075 17185 tablet_bootstrap.cc:654] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:36.707381 16979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af54823c7c644c92b460fde468cdfaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.714007 17183 tablet_bootstrap.cc:654] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:36.713835 17193 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:36.714637 17193 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.717499 17193 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:36.721330 17184 tablet_bootstrap.cc:654] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:36.736475 17192 raft_consensus.cc:493] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:36.737289 17192 raft_consensus.cc:515] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:36.754526 17186 ts_tablet_manager.cc:1434] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.234s	user 0.007s	sys 0.026s
I20260709 13:05:36.755694 17186 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:36.742175 16978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cf831311924bb7b8afbc30043e1550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.766232 17192 leader_election.cc:290] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:36.769066 16937 debug-util.cc:398] Leaking SignalData structure 0x7b0800320c00 after lost signal to thread 7484
W20260709 13:05:36.770280 16937 debug-util.cc:398] Leaking SignalData structure 0x7b080024c840 after lost signal to thread 16848
I20260709 13:05:36.760921 17052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b68459f0de48d98ad40888859c37f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.800608 16973 tablet_service.cc:1511] Processing CreateTablet for tablet 9c2aedbc9c254811b0e3bb47679bb443 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:05:36.802399 16973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2aedbc9c254811b0e3bb47679bb443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.806243 17128 tablet_service.cc:1511] Processing CreateTablet for tablet 61b68459f0de48d98ad40888859c37f5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:05:36.807852 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b68459f0de48d98ad40888859c37f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.825699 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:36.828158 17102 leader_election.cc:343] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:36.834115 17047 tablet_service.cc:1511] Processing CreateTablet for tablet f4cf831311924bb7b8afbc30043e1550 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:05:36.834609 17046 tablet_service.cc:1511] Processing CreateTablet for tablet 9c2aedbc9c254811b0e3bb47679bb443 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:05:36.836297 17047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cf831311924bb7b8afbc30043e1550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.838548 17127 tablet_service.cc:1511] Processing CreateTablet for tablet f4cf831311924bb7b8afbc30043e1550 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:05:36.840276 17127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cf831311924bb7b8afbc30043e1550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.844420 17183 tablet_bootstrap.cc:492] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:36.845147 17183 ts_tablet_manager.cc:1403] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.242s	user 0.014s	sys 0.002s
I20260709 13:05:36.848765 17183 raft_consensus.cc:359] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:36.849849 17183 raft_consensus.cc:385] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:36.850307 17046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2aedbc9c254811b0e3bb47679bb443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.850205 17183 raft_consensus.cc:740] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:36.857115 17121 tablet_service.cc:1511] Processing CreateTablet for tablet d4a02d063e934712ace429f988380d0f (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:05:36.858363 17185 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:36.858749 17121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4a02d063e934712ace429f988380d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.858980 17185 ts_tablet_manager.cc:1403] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.273s	user 0.017s	sys 0.005s
I20260709 13:05:36.861567 17186 tablet_bootstrap.cc:654] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:36.871838 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:36.878371 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:36.879326 16916 raft_consensus.cc:2468] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:36.880828 17105 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:36.880709 17185 raft_consensus.cc:359] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:36.881757 17185 raft_consensus.cc:385] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:36.881865 17191 raft_consensus.cc:2804] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:36.882150 17185 raft_consensus.cc:740] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:36.882369 17191 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:36.882848 17191 raft_consensus.cc:3060] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:36.891682 17191 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:36.897609 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:36.898355 16916 raft_consensus.cc:3060] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:36.893047 17185 consensus_queue.cc:260] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:36.910363 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:36.912864 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
W20260709 13:05:36.922812 16954 leader_election.cc:343] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:36.904592 17191 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:36.930197 16975 tablet_service.cc:1511] Processing CreateTablet for tablet d4a02d063e934712ace429f988380d0f (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:05:36.932724 17054 tablet_service.cc:1511] Processing CreateTablet for tablet 17f3f3ca934e4998ab856ebde5eda039 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:05:36.933367 17125 tablet_service.cc:1511] Processing CreateTablet for tablet 17f3f3ca934e4998ab856ebde5eda039 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:05:36.941607 16975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4a02d063e934712ace429f988380d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.942082 17125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f3f3ca934e4998ab856ebde5eda039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.941607 17054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f3f3ca934e4998ab856ebde5eda039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.919028 17183 consensus_queue.cc:260] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:36.947679 17183 ts_tablet_manager.cc:1434] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.102s	user 0.007s	sys 0.000s
I20260709 13:05:36.922176 16916 raft_consensus.cc:2468] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:36.954707 17055 tablet_service.cc:1511] Processing CreateTablet for tablet db95bd0ca80745f09d5104034b88d18c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:05:36.956168 17055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db95bd0ca80745f09d5104034b88d18c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:05:36.940438 17102 leader_election.cc:343] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:36.960592 17105 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:36.964572 17184 tablet_bootstrap.cc:492] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:36.965199 17184 ts_tablet_manager.cc:1403] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.303s	user 0.012s	sys 0.009s
I20260709 13:05:36.968757 16972 tablet_service.cc:1511] Processing CreateTablet for tablet db95bd0ca80745f09d5104034b88d18c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:05:36.968451 17184 raft_consensus.cc:359] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:36.970130 17184 raft_consensus.cc:385] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:36.970253 16972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db95bd0ca80745f09d5104034b88d18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.970506 17184 raft_consensus.cc:740] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:36.977698 17184 consensus_queue.cc:260] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:36.981464 17191 raft_consensus.cc:2804] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:36.983978 17185 ts_tablet_manager.cc:1434] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.125s	user 0.010s	sys 0.000s
I20260709 13:05:36.987442 17185 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:36.988628 17124 tablet_service.cc:1511] Processing CreateTablet for tablet db95bd0ca80745f09d5104034b88d18c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:05:36.990111 17124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db95bd0ca80745f09d5104034b88d18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.991083 17186 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:36.991660 17186 ts_tablet_manager.cc:1403] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.236s	user 0.006s	sys 0.020s
I20260709 13:05:36.993918 17183 tablet_bootstrap.cc:492] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:36.996579 17046 tablet_service.cc:1511] Processing CreateTablet for tablet 73e77860bb944a5e97539267e7d0f9ba (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:05:36.998255 17046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e77860bb944a5e97539267e7d0f9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:36.999162 17198 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.000465 17198 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.003830 16977 tablet_service.cc:1511] Processing CreateTablet for tablet 73e77860bb944a5e97539267e7d0f9ba (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:05:37.006104 16977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e77860bb944a5e97539267e7d0f9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.031742 17198 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:37.041975 16978 tablet_service.cc:1511] Processing CreateTablet for tablet e4a541f57ac040ee9de0e73269605ab5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:05:37.043550 16978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4a541f57ac040ee9de0e73269605ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.047372 17184 ts_tablet_manager.cc:1434] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.082s	user 0.001s	sys 0.005s
I20260709 13:05:37.049736 17186 raft_consensus.cc:359] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.050855 17186 raft_consensus.cc:385] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.051172 17186 raft_consensus.cc:740] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.051961 17186 consensus_queue.cc:260] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.053884 16979 tablet_service.cc:1511] Processing CreateTablet for tablet 7b240339eb134181914367d7072b86b6 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:05:37.055370 16979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b240339eb134181914367d7072b86b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.056519 17121 tablet_service.cc:1511] Processing CreateTablet for tablet e4a541f57ac040ee9de0e73269605ab5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:05:37.057916 17121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4a541f57ac040ee9de0e73269605ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.060819 16973 tablet_service.cc:1511] Processing CreateTablet for tablet 696628411ba54b68a0ae0c4b77350e50 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:05:37.062388 16973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 696628411ba54b68a0ae0c4b77350e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.066864 17126 tablet_service.cc:1511] Processing CreateTablet for tablet e072a4fb708444f6ade1000459bc8c67 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:05:37.068274 17126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e072a4fb708444f6ade1000459bc8c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.069015 17183 tablet_bootstrap.cc:654] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.081822 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:37.083180 16980 tablet_service.cc:1511] Processing CreateTablet for tablet d8c8535d6c24400ab8efb3d995b908e6 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
W20260709 13:05:37.083555 16957 leader_election.cc:343] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.084101 16957 leader_election.cc:304] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:37.084566 16980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8c8535d6c24400ab8efb3d995b908e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.085330 17192 raft_consensus.cc:2749] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.097829 17184 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:37.098213 16905 tablet_service.cc:1511] Processing CreateTablet for tablet 9c2aedbc9c254811b0e3bb47679bb443 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:05:37.099812 16905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2aedbc9c254811b0e3bb47679bb443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.103454 16897 tablet_service.cc:1511] Processing CreateTablet for tablet d4a02d063e934712ace429f988380d0f (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:05:37.105698 16897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4a02d063e934712ace429f988380d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.109735 17185 tablet_bootstrap.cc:654] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.110909 16981 tablet_service.cc:1511] Processing CreateTablet for tablet ea9f0699e38347ebaa746bfc5f3f1789 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:05:37.112411 16981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9f0699e38347ebaa746bfc5f3f1789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.118952 17125 tablet_service.cc:1511] Processing CreateTablet for tablet 7b240339eb134181914367d7072b86b6 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:05:37.120492 17125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b240339eb134181914367d7072b86b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.127276 16904 tablet_service.cc:1511] Processing CreateTablet for tablet 17f3f3ca934e4998ab856ebde5eda039 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:05:37.128881 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f3f3ca934e4998ab856ebde5eda039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.128837 17129 tablet_service.cc:1511] Processing CreateTablet for tablet d8c8535d6c24400ab8efb3d995b908e6 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:05:37.130491 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8c8535d6c24400ab8efb3d995b908e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.131356 17184 tablet_bootstrap.cc:654] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.133777 16899 tablet_service.cc:1511] Processing CreateTablet for tablet 73e77860bb944a5e97539267e7d0f9ba (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:05:37.134498 17123 tablet_service.cc:1511] Processing CreateTablet for tablet e8c8eadfc13d42fc83c0bbdfbc0a98dd (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:05:37.135154 16899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e77860bb944a5e97539267e7d0f9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.135752 17123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c8eadfc13d42fc83c0bbdfbc0a98dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.138547 16979 tablet_service.cc:1511] Processing CreateTablet for tablet 34c0447f54954c6b9ad3b7eb4f9402fa (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:05:37.140162 16979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c0447f54954c6b9ad3b7eb4f9402fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.146858 17128 tablet_service.cc:1511] Processing CreateTablet for tablet ea9f0699e38347ebaa746bfc5f3f1789 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:05:37.148372 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9f0699e38347ebaa746bfc5f3f1789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.159644 17122 tablet_service.cc:1511] Processing CreateTablet for tablet 3f5d356796924994baa41134536450d2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:05:37.160183 16977 tablet_service.cc:1511] Processing CreateTablet for tablet 3f5d356796924994baa41134536450d2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:05:37.161131 17122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5d356796924994baa41134536450d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.184899 17183 tablet_bootstrap.cc:492] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:37.187784 17052 tablet_service.cc:1511] Processing CreateTablet for tablet 7b240339eb134181914367d7072b86b6 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:05:37.198518 16900 tablet_service.cc:1511] Processing CreateTablet for tablet e072a4fb708444f6ade1000459bc8c67 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:05:37.198678 16901 tablet_service.cc:1511] Processing CreateTablet for tablet 696628411ba54b68a0ae0c4b77350e50 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:05:37.199864 16900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e072a4fb708444f6ade1000459bc8c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.200515 16901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 696628411ba54b68a0ae0c4b77350e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.202029 17052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b240339eb134181914367d7072b86b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.204881 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:37.206771 17028 leader_election.cc:343] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.206840 16906 tablet_service.cc:1511] Processing CreateTablet for tablet d8c8535d6c24400ab8efb3d995b908e6 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:05:37.208397 16906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8c8535d6c24400ab8efb3d995b908e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.171119 16977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5d356796924994baa41134536450d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.210983 17127 tablet_service.cc:1511] Processing CreateTablet for tablet 64212ad563c34c83812b266155d7086a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:05:37.188032 17049 tablet_service.cc:1511] Processing CreateTablet for tablet 696628411ba54b68a0ae0c4b77350e50 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:05:37.212321 17049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 696628411ba54b68a0ae0c4b77350e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.214085 17186 ts_tablet_manager.cc:1434] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.222s	user 0.010s	sys 0.001s
I20260709 13:05:37.217207 17186 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:37.188902 17050 tablet_service.cc:1511] Processing CreateTablet for tablet ea9f0699e38347ebaa746bfc5f3f1789 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:05:37.218286 17050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9f0699e38347ebaa746bfc5f3f1789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.189452 17048 tablet_service.cc:1511] Processing CreateTablet for tablet e8c8eadfc13d42fc83c0bbdfbc0a98dd (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:05:37.222215 17048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c8eadfc13d42fc83c0bbdfbc0a98dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.190603 17051 tablet_service.cc:1511] Processing CreateTablet for tablet 34c0447f54954c6b9ad3b7eb4f9402fa (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:05:37.190604 17047 tablet_service.cc:1511] Processing CreateTablet for tablet e072a4fb708444f6ade1000459bc8c67 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:05:37.196071 16902 tablet_service.cc:1511] Processing CreateTablet for tablet e4a541f57ac040ee9de0e73269605ab5 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:05:37.226277 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c0447f54954c6b9ad3b7eb4f9402fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.226593 16902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4a541f57ac040ee9de0e73269605ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.233314 17047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e072a4fb708444f6ade1000459bc8c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.234889 16899 tablet_service.cc:1511] Processing CreateTablet for tablet 34c0447f54954c6b9ad3b7eb4f9402fa (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:05:37.236295 16899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c0447f54954c6b9ad3b7eb4f9402fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.212321 17127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64212ad563c34c83812b266155d7086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.243012 16903 tablet_service.cc:1511] Processing CreateTablet for tablet 3f5d356796924994baa41134536450d2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:05:37.244601 16903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5d356796924994baa41134536450d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.254293 16898 tablet_service.cc:1511] Processing CreateTablet for tablet 64212ad563c34c83812b266155d7086a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:05:37.255455 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:37.256183 17065 raft_consensus.cc:2468] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:37.256369 16898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64212ad563c34c83812b266155d7086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.256210 16973 tablet_service.cc:1511] Processing CreateTablet for tablet df062cbe424f401ea234f20cdb26eff3 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:05:37.258540 16973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df062cbe424f401ea234f20cdb26eff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.258347 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:37.261958 16879 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:37.262883 17193 raft_consensus.cc:2764] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:05:37.268215 16904 tablet_service.cc:1511] Processing CreateTablet for tablet e8c8eadfc13d42fc83c0bbdfbc0a98dd (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:05:37.269925 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c8eadfc13d42fc83c0bbdfbc0a98dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.280526 17186 tablet_bootstrap.cc:654] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.291504 17054 tablet_service.cc:1511] Processing CreateTablet for tablet 64212ad563c34c83812b266155d7086a (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:05:37.171656 17191 raft_consensus.cc:697] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:37.297423 17055 tablet_service.cc:1511] Processing CreateTablet for tablet df062cbe424f401ea234f20cdb26eff3 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:05:37.305296 17054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64212ad563c34c83812b266155d7086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.308511 17055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df062cbe424f401ea234f20cdb26eff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.286546 17183 ts_tablet_manager.cc:1403] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.293s	user 0.015s	sys 0.001s
I20260709 13:05:37.313544 17183 raft_consensus.cc:359] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.314289 17183 raft_consensus.cc:385] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.314590 17183 raft_consensus.cc:740] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.320413 17193 raft_consensus.cc:493] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.321159 17193 raft_consensus.cc:515] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.321727 17183 consensus_queue.cc:260] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.324122 17185 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:37.324687 17185 ts_tablet_manager.cc:1403] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.337s	user 0.013s	sys 0.013s
I20260709 13:05:37.325801 17193 leader_election.cc:290] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:37.332757 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:37.294216 17191 consensus_queue.cc:237] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:37.334200 16879 leader_election.cc:343] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.339833 17192 raft_consensus.cc:493] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.340425 17192 raft_consensus.cc:515] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.342935 17192 leader_election.cc:290] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:37.348193 17185 raft_consensus.cc:359] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.349035 17185 raft_consensus.cc:385] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.350967 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:37.349356 17185 raft_consensus.cc:740] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.356577 17211 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.357115 17211 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.359481 17211 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:37.355646 17185 consensus_queue.cc:260] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.360953 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:37.361814 17185 ts_tablet_manager.cc:1434] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
W20260709 13:05:37.362692 17102 leader_election.cc:343] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.361048 16901 tablet_service.cc:1511] Processing CreateTablet for tablet aa31fcff86ca4af6b0cb7c4be6736c57 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260709 13:05:37.363533 16954 leader_election.cc:343] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.370498 17183 ts_tablet_manager.cc:1434] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.060s	user 0.007s	sys 0.000s
I20260709 13:05:37.371575 17183 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:37.374342 16901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa31fcff86ca4af6b0cb7c4be6736c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.377600 17183 tablet_bootstrap.cc:654] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.385576 17185 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:37.387063 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:37.389369 17031 leader_election.cc:343] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.390013 17031 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:37.394110 17123 tablet_service.cc:1511] Processing CreateTablet for tablet df062cbe424f401ea234f20cdb26eff3 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:05:37.395473 17123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df062cbe424f401ea234f20cdb26eff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.397653 17198 raft_consensus.cc:2749] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.421295 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:37.422745 16954 leader_election.cc:343] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.423228 16954 leader_election.cc:304] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:37.429286 17128 tablet_service.cc:1511] Processing CreateTablet for tablet bda4b312c29c472f8c19d32d3029ee76 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:05:37.430660 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda4b312c29c472f8c19d32d3029ee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.425590 17192 raft_consensus.cc:2749] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.435315 17046 tablet_service.cc:1511] Processing CreateTablet for tablet 7ee862553fe342db878b5b9582fb233c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:05:37.435900 17124 tablet_service.cc:1511] Processing CreateTablet for tablet 7ee862553fe342db878b5b9582fb233c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:05:37.437222 17124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ee862553fe342db878b5b9582fb233c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.439250 17046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ee862553fe342db878b5b9582fb233c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.446599 16973 tablet_service.cc:1511] Processing CreateTablet for tablet aa31fcff86ca4af6b0cb7c4be6736c57 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:05:37.447993 16973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa31fcff86ca4af6b0cb7c4be6736c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.448560 16903 tablet_service.cc:1511] Processing CreateTablet for tablet bda4b312c29c472f8c19d32d3029ee76 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:05:37.458230 16977 tablet_service.cc:1511] Processing CreateTablet for tablet bda4b312c29c472f8c19d32d3029ee76 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:05:37.459679 16977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda4b312c29c472f8c19d32d3029ee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.465940 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:37.465366 17185 tablet_bootstrap.cc:654] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:37.484802 16881 leader_election.cc:343] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.485486 16881 leader_election.cc:304] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:37.486949 17193 raft_consensus.cc:2749] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.475158 17183 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:37.491845 17183 ts_tablet_manager.cc:1403] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.120s	user 0.009s	sys 0.029s
I20260709 13:05:37.495108 17183 raft_consensus.cc:359] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.495865 17183 raft_consensus.cc:385] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.496129 17193 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.496655 17193 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.499318 17193 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:37.499560 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:37.500984 17186 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:37.504936 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:37.506918 17104 leader_election.cc:343] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.507404 17104 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:37.507742 17055 tablet_service.cc:1511] Processing CreateTablet for tablet 2517faf22ec44930814616834f5c6b19 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:05:37.508538 17211 raft_consensus.cc:2749] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.509126 17055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2517faf22ec44930814616834f5c6b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.510768 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:37.510962 17123 tablet_service.cc:1511] Processing CreateTablet for tablet e5e73389947c4fad9319efe807e8016c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260709 13:05:37.502301 16801 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 47564.1us
I20260709 13:05:37.512315 17123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e73389947c4fad9319efe807e8016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.479562 16903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda4b312c29c472f8c19d32d3029ee76. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:05:37.502741 16800 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 46123.4us
I20260709 13:05:37.466372 17048 tablet_service.cc:1511] Processing CreateTablet for tablet aa31fcff86ca4af6b0cb7c4be6736c57 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260709 13:05:37.501286 16879 leader_election.cc:343] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.527151 16981 tablet_service.cc:1511] Processing CreateTablet for tablet 5aa0046c7a4f453c8c3f27e71c92fbe2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:05:37.527733 16898 tablet_service.cc:1511] Processing CreateTablet for tablet 6e5613a2ba864f379a89f716babe1879 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:05:37.528523 16981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aa0046c7a4f453c8c3f27e71c92fbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.528369 16899 tablet_service.cc:1511] Processing CreateTablet for tablet 7ee862553fe342db878b5b9582fb233c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:05:37.528975 16898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5613a2ba864f379a89f716babe1879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.529634 16904 tablet_service.cc:1511] Processing CreateTablet for tablet b78448a359344202ad036ddd40ccee93 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:05:37.530731 16902 tablet_service.cc:1511] Processing CreateTablet for tablet 2517faf22ec44930814616834f5c6b19 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:05:37.533010 16901 tablet_service.cc:1511] Processing CreateTablet for tablet 5aa0046c7a4f453c8c3f27e71c92fbe2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:05:37.516253 16978 tablet_service.cc:1511] Processing CreateTablet for tablet b78448a359344202ad036ddd40ccee93 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:05:37.496148 17183 raft_consensus.cc:740] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.535190 16978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78448a359344202ad036ddd40ccee93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.535089 17183 consensus_queue.cc:260] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.478749 17184 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:37.518489 16980 tablet_service.cc:1511] Processing CreateTablet for tablet 6e5613a2ba864f379a89f716babe1879 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:05:37.520795 17049 tablet_service.cc:1511] Processing CreateTablet for tablet 11c63f7d1e40424ea2d7f83316591ac2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:05:37.537768 17184 ts_tablet_manager.cc:1403] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.440s	user 0.025s	sys 0.055s
I20260709 13:05:37.538407 17049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c63f7d1e40424ea2d7f83316591ac2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:05:37.538334 16881 leader_election.cc:343] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.539271 16980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5613a2ba864f379a89f716babe1879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.539800 16881 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:37.523106 17124 tablet_service.cc:1511] Processing CreateTablet for tablet 11c63f7d1e40424ea2d7f83316591ac2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:05:37.522549 16813 catalog_manager.cc:5697] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:37.541836 17183 ts_tablet_manager.cc:1434] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.050s	user 0.006s	sys 0.002s
I20260709 13:05:37.542296 17124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c63f7d1e40424ea2d7f83316591ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.542940 17183 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:37.523878 16975 tablet_service.cc:1511] Processing CreateTablet for tablet 11c63f7d1e40424ea2d7f83316591ac2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:05:37.545100 16975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c63f7d1e40424ea2d7f83316591ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.541326 17184 raft_consensus.cc:359] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.546257 17184 raft_consensus.cc:385] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.546339 17193 raft_consensus.cc:2749] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.505030 17186 ts_tablet_manager.cc:1403] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.288s	user 0.017s	sys 0.003s
I20260709 13:05:37.529659 16899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ee862553fe342db878b5b9582fb233c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.549151 17184 raft_consensus.cc:740] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.550241 17184 consensus_queue.cc:260] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.552798 17184 ts_tablet_manager.cc:1434] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:05:37.553887 17184 tablet_bootstrap.cc:492] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:37.557796 16904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78448a359344202ad036ddd40ccee93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.558259 17048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa31fcff86ca4af6b0cb7c4be6736c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.560839 16901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aa0046c7a4f453c8c3f27e71c92fbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.560585 17186 raft_consensus.cc:359] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.561475 17186 raft_consensus.cc:385] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.561635 16902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2517faf22ec44930814616834f5c6b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.561818 17186 raft_consensus.cc:740] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.536952 17128 tablet_service.cc:1511] Processing CreateTablet for tablet 6e5613a2ba864f379a89f716babe1879 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:05:37.577322 17052 tablet_service.cc:1511] Processing CreateTablet for tablet 5aa0046c7a4f453c8c3f27e71c92fbe2 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:05:37.578771 17128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5613a2ba864f379a89f716babe1879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.578771 17052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aa0046c7a4f453c8c3f27e71c92fbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.537027 17046 tablet_service.cc:1511] Processing CreateTablet for tablet e5e73389947c4fad9319efe807e8016c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:05:37.582625 17046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e73389947c4fad9319efe807e8016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.587239 16897 tablet_service.cc:1511] Processing CreateTablet for tablet 9fcfa9f9a5224b69b785589ce292ca3e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:05:37.588053 16972 tablet_service.cc:1511] Processing CreateTablet for tablet 9fcfa9f9a5224b69b785589ce292ca3e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:05:37.588690 16897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcfa9f9a5224b69b785589ce292ca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.589299 16972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcfa9f9a5224b69b785589ce292ca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.587246 17125 tablet_service.cc:1511] Processing CreateTablet for tablet 9fcfa9f9a5224b69b785589ce292ca3e (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:05:37.591006 17125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcfa9f9a5224b69b785589ce292ca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.577819 17183 tablet_bootstrap.cc:654] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.583105 17127 tablet_service.cc:1511] Processing CreateTablet for tablet 2517faf22ec44930814616834f5c6b19 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:05:37.599890 16979 tablet_service.cc:1511] Processing CreateTablet for tablet e5e73389947c4fad9319efe807e8016c (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:05:37.602476 17054 tablet_service.cc:1511] Processing CreateTablet for tablet 9b961524f09a42878311bbfebdbf23de (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:05:37.604126 17054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b961524f09a42878311bbfebdbf23de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.606295 17047 tablet_service.cc:1511] Processing CreateTablet for tablet b78448a359344202ad036ddd40ccee93 (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:05:37.607692 17047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78448a359344202ad036ddd40ccee93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.613318 17129 tablet_service.cc:1511] Processing CreateTablet for tablet 9b961524f09a42878311bbfebdbf23de (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:05:37.618086 17127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2517faf22ec44930814616834f5c6b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.619498 17129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b961524f09a42878311bbfebdbf23de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.623102 17185 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:37.623693 17185 ts_tablet_manager.cc:1403] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.238s	user 0.015s	sys 0.002s
I20260709 13:05:37.623957 17184 tablet_bootstrap.cc:654] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.601239 16979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e73389947c4fad9319efe807e8016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.636777 16905 tablet_service.cc:1511] Processing CreateTablet for tablet 9b961524f09a42878311bbfebdbf23de (DEFAULT_TABLE table=test-workload [id=cf4aa2813212419bb62211da2ffadbab]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:05:37.638221 16905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b961524f09a42878311bbfebdbf23de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:37.597383 17186 consensus_queue.cc:260] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.645117 17198 raft_consensus.cc:493] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.647758 17198 raft_consensus.cc:515] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.665707 17185 raft_consensus.cc:359] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.675117 17198 leader_election.cc:290] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:37.679949 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:37.680722 16916 raft_consensus.cc:2468] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:37.685055 17031 leader_election.cc:304] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:37.694233 17191 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.694905 17191 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.697346 17192 raft_consensus.cc:493] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.697681 17198 raft_consensus.cc:2804] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:37.698138 17198 raft_consensus.cc:493] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:37.697933 17192 raft_consensus.cc:515] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.698599 17198 raft_consensus.cc:3060] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:37.698987 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:37.700201 17192 leader_election.cc:290] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:37.701743 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:37.702646 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:37.682618 17185 raft_consensus.cc:385] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:05:37.703073 17104 leader_election.cc:343] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.698210 17191 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:37.712118 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:37.712671 16957 leader_election.cc:343] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:37.717090 16954 leader_election.cc:343] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.717592 16954 leader_election.cc:304] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:37.718354 17192 raft_consensus.cc:2749] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:05:37.714596 17102 leader_election.cc:343] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.719471 17186 ts_tablet_manager.cc:1434] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.162s	user 0.014s	sys 0.000s
I20260709 13:05:37.719419 17102 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:37.720680 17211 raft_consensus.cc:2749] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.721921 17186 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:37.704274 17185 raft_consensus.cc:740] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.733604 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:37.737035 17030 leader_election.cc:343] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.744095 17186 tablet_bootstrap.cc:654] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.747704 17198 raft_consensus.cc:515] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.737564 17185 consensus_queue.cc:260] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.754259 17185 ts_tablet_manager.cc:1434] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.130s	user 0.003s	sys 0.008s
I20260709 13:05:37.755388 17185 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:37.758219 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:37.758988 16916 raft_consensus.cc:3060] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:37.766593 16916 raft_consensus.cc:2468] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:37.767995 17031 leader_election.cc:304] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:37.768559 17183 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:37.769102 17183 ts_tablet_manager.cc:1403] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.226s	user 0.015s	sys 0.013s
I20260709 13:05:37.770924 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
W20260709 13:05:37.772320 17030 leader_election.cc:343] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.773545 17227 raft_consensus.cc:2804] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:37.774470 17198 leader_election.cc:290] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:37.780206 17186 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:37.780857 17186 ts_tablet_manager.cc:1403] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.059s	user 0.006s	sys 0.007s
I20260709 13:05:37.783654 17184 tablet_bootstrap.cc:492] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:37.784242 17184 ts_tablet_manager.cc:1403] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.231s	user 0.016s	sys 0.008s
I20260709 13:05:37.785856 17186 raft_consensus.cc:359] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.786701 17186 raft_consensus.cc:385] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.786993 17186 raft_consensus.cc:740] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.787742 17186 consensus_queue.cc:260] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.788371 17183 raft_consensus.cc:359] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.789083 17183 raft_consensus.cc:385] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.795976 17198 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.789391 17183 raft_consensus.cc:740] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.799491 17198 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.800545 17183 consensus_queue.cc:260] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.801678 17198 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:37.802515 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:37.803941 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:37.804226 17183 ts_tablet_manager.cc:1434] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 13:05:37.805660 17183 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:37.798565 17227 raft_consensus.cc:697] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:37.806937 17227 consensus_queue.cc:237] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:37.804075 17031 leader_election.cc:343] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:37.810189 17030 leader_election.cc:343] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:37.790278 17186 ts_tablet_manager.cc:1434] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.004s
I20260709 13:05:37.796855 17185 tablet_bootstrap.cc:654] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.811802 17186 tablet_bootstrap.cc:492] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:37.813323 17030 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:37.814751 17198 raft_consensus.cc:2749] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:37.821151 17183 tablet_bootstrap.cc:654] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.822619 17186 tablet_bootstrap.cc:654] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.796514 17184 raft_consensus.cc:359] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.826577 17184 raft_consensus.cc:385] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.826974 17184 raft_consensus.cc:740] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.828256 17184 consensus_queue.cc:260] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.835005 17184 ts_tablet_manager.cc:1434] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.050s	user 0.010s	sys 0.000s
I20260709 13:05:37.826608 17185 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:37.835971 16813 catalog_manager.cc:5697] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:37.842890 17183 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:37.843149 17185 ts_tablet_manager.cc:1403] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.088s	user 0.008s	sys 0.014s
I20260709 13:05:37.843557 17183 ts_tablet_manager.cc:1403] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.012s
I20260709 13:05:37.845968 17186 tablet_bootstrap.cc:492] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:37.846561 17186 ts_tablet_manager.cc:1403] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.035s	user 0.018s	sys 0.007s
I20260709 13:05:37.847282 17183 raft_consensus.cc:359] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.848129 17183 raft_consensus.cc:385] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.848496 17183 raft_consensus.cc:740] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.849298 17183 consensus_queue.cc:260] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.850201 17186 raft_consensus.cc:359] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.851013 17186 raft_consensus.cc:385] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.851333 17186 raft_consensus.cc:740] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.852046 17183 ts_tablet_manager.cc:1434] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.003s
I20260709 13:05:37.852151 17186 consensus_queue.cc:260] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.853536 17183 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:37.857141 17184 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:37.861871 17183 tablet_bootstrap.cc:654] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.866888 17185 raft_consensus.cc:359] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.867818 17185 raft_consensus.cc:385] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.868165 17185 raft_consensus.cc:740] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.869175 17185 consensus_queue.cc:260] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.870072 17186 ts_tablet_manager.cc:1434] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.001s
I20260709 13:05:37.871246 17186 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:37.874615 17183 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:37.875252 17183 ts_tablet_manager.cc:1403] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.007s
I20260709 13:05:37.878772 17185 ts_tablet_manager.cc:1434] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.001s
W20260709 13:05:37.879030 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:37.878906 17183 raft_consensus.cc:359] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.880223 17184 tablet_bootstrap.cc:654] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.880498 17183 raft_consensus.cc:385] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.880946 17183 raft_consensus.cc:740] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.881927 17186 tablet_bootstrap.cc:654] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.881942 17183 consensus_queue.cc:260] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.885217 17183 ts_tablet_manager.cc:1434] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 13:05:37.886672 17183 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:37.886988 17185 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:37.894307 17183 tablet_bootstrap.cc:654] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.896690 17184 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:37.897348 17184 ts_tablet_manager.cc:1403] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.040s	user 0.013s	sys 0.000s
I20260709 13:05:37.901521 17184 raft_consensus.cc:359] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.902437 17184 raft_consensus.cc:385] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.902802 17184 raft_consensus.cc:740] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.903805 17184 consensus_queue.cc:260] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.904655 17185 tablet_bootstrap.cc:654] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.916846 17184 ts_tablet_manager.cc:1434] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 13:05:37.922940 17184 tablet_bootstrap.cc:492] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:37.928879 17183 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:37.929536 17183 ts_tablet_manager.cc:1403] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.043s	user 0.008s	sys 0.010s
I20260709 13:05:37.932852 17183 raft_consensus.cc:359] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.934859 17185 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:37.934913 17183 raft_consensus.cc:385] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.935755 17185 ts_tablet_manager.cc:1403] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.049s	user 0.016s	sys 0.000s
I20260709 13:05:37.935772 17183 raft_consensus.cc:740] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.936290 17186 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:37.936884 17186 ts_tablet_manager.cc:1403] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.066s	user 0.019s	sys 0.029s
I20260709 13:05:37.937186 17183 consensus_queue.cc:260] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.938743 17184 tablet_bootstrap.cc:654] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.939642 17185 raft_consensus.cc:359] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.940528 17185 raft_consensus.cc:385] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.941094 17185 raft_consensus.cc:740] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.940537 17186 raft_consensus.cc:359] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.941696 17186 raft_consensus.cc:385] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.942049 17186 raft_consensus.cc:740] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.942232 17185 consensus_queue.cc:260] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.942991 17186 consensus_queue.cc:260] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:37.948225 17183 ts_tablet_manager.cc:1434] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 13:05:37.949524 17183 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:37.954325 17186 ts_tablet_manager.cc:1434] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:05:37.956324 17183 tablet_bootstrap.cc:654] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.968145 17183 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:37.967689 17191 consensus_queue.cc:1048] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:37.968935 17183 ts_tablet_manager.cc:1403] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.007s
I20260709 13:05:37.967602 17185 ts_tablet_manager.cc:1434] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.004s
I20260709 13:05:37.971527 17185 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:37.972584 17183 raft_consensus.cc:359] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.973491 17183 raft_consensus.cc:385] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:37.973852 17183 raft_consensus.cc:740] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:37.974819 17183 consensus_queue.cc:260] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:37.982028 17065 raft_consensus.cc:3060] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:37.982942 17186 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:37.989490 17211 raft_consensus.cc:493] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.989580 17227 raft_consensus.cc:493] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:37.990221 17211 raft_consensus.cc:515] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:37.990417 17227 raft_consensus.cc:515] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:37.994339 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:37.994971 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:37.995836 16990 raft_consensus.cc:2468] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:37.997485 17030 leader_election.cc:304] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:37.998039 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:37.998338 17186 tablet_bootstrap.cc:654] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:37.998723 17198 raft_consensus.cc:2804] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:37.999191 17198 raft_consensus.cc:493] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:37.999637 17198 raft_consensus.cc:3060] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:38.000180 17105 leader_election.cc:343] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.002028 17211 leader_election.cc:290] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.004293 17185 tablet_bootstrap.cc:654] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.009523 17227 leader_election.cc:290] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:38.001981 17104 leader_election.cc:343] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.010740 17104 leader_election.cc:304] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:38.012763 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:38.013784 17246 raft_consensus.cc:2749] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.015244 17184 tablet_bootstrap.cc:492] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.015875 17184 ts_tablet_manager.cc:1403] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.093s	user 0.032s	sys 0.017s
W20260709 13:05:38.018226 17028 leader_election.cc:343] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.019622 17186 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:38.020359 17186 ts_tablet_manager.cc:1403] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.038s	user 0.016s	sys 0.003s
I20260709 13:05:38.019950 17184 raft_consensus.cc:359] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.020982 17184 raft_consensus.cc:385] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.021384 17184 raft_consensus.cc:740] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.022579 17184 consensus_queue.cc:260] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.023804 17211 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.024166 17186 raft_consensus.cc:359] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.025030 17186 raft_consensus.cc:385] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.024494 17211 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.025388 17186 raft_consensus.cc:740] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.026391 17185 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:38.026499 17186 consensus_queue.cc:260] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.027464 17193 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.028033 17193 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.030385 17193 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.030651 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.031682 17185 ts_tablet_manager.cc:1403] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.056s	user 0.011s	sys 0.004s
I20260709 13:05:38.032444 17186 ts_tablet_manager.cc:1434] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.001s
I20260709 13:05:38.032979 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:38.033793 17186 tablet_bootstrap.cc:492] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:38.033804 17139 raft_consensus.cc:2468] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:38.034199 17184 ts_tablet_manager.cc:1434] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.001s
I20260709 13:05:38.035290 17185 raft_consensus.cc:359] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
W20260709 13:05:38.035828 17102 leader_election.cc:343] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.036149 17185 raft_consensus.cc:385] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.036574 17185 raft_consensus.cc:740] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.036785 16879 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:38.037953 17193 raft_consensus.cc:2804] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.038488 17193 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.038941 17193 raft_consensus.cc:3060] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.041975 17184 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.041813 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.045023 17186 tablet_bootstrap.cc:654] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.045961 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:38.047979 16879 leader_election.cc:343] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.048616 17211 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.053081 17211 consensus_queue.cc:1048] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:38.048162 17183 ts_tablet_manager.cc:1434] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.079s	user 0.006s	sys 0.000s
I20260709 13:05:38.060822 17192 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.061513 17192 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.064033 17192 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.037554 17185 consensus_queue.cc:260] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.050746 17198 raft_consensus.cc:515] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.067991 17227 raft_consensus.cc:493] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.068838 17227 raft_consensus.cc:515] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.072163 17184 tablet_bootstrap.cc:654] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.072803 17198 leader_election.cc:290] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.074180 17227 leader_election.cc:290] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.073102 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:38.077083 17031 leader_election.cc:343] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.066968 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:38.079743 17139 raft_consensus.cc:2468] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:38.079761 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:38.081337 16954 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:38.066735 17183 tablet_bootstrap.cc:492] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
W20260709 13:05:38.081570 17028 leader_election.cc:343] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.082839 17192 raft_consensus.cc:2804] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.083470 17192 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.084020 17192 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.090278 17183 tablet_bootstrap.cc:654] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.091445 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:38.092207 16990 raft_consensus.cc:3060] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.095368 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.096237 16991 raft_consensus.cc:2468] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:38.098577 17030 leader_election.cc:304] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:38.099896 17184 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.100559 17184 ts_tablet_manager.cc:1403] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.059s	user 0.012s	sys 0.003s
I20260709 13:05:38.101899 17183 tablet_bootstrap.cc:492] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:38.102478 17183 ts_tablet_manager.cc:1403] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.042s	user 0.014s	sys 0.000s
I20260709 13:05:38.104240 17184 raft_consensus.cc:359] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.067353 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.105167 17184 raft_consensus.cc:385] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.105752 17184 raft_consensus.cc:740] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.105494 17183 raft_consensus.cc:359] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.106501 17183 raft_consensus.cc:385] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.106796 17183 raft_consensus.cc:740] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.106848 17184 consensus_queue.cc:260] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:38.107194 16954 leader_election.cc:343] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.107591 17183 consensus_queue.cc:260] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.109935 17211 raft_consensus.cc:493] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.110468 17211 raft_consensus.cc:515] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.112295 17227 raft_consensus.cc:2804] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.112787 17227 raft_consensus.cc:493] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.113039 17211 leader_election.cc:290] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.113222 17227 raft_consensus.cc:3060] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.124994 17184 ts_tablet_manager.cc:1434] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.004s
I20260709 13:05:38.114634 17183 ts_tablet_manager.cc:1434] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 13:05:38.115944 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.125908 16915 raft_consensus.cc:2468] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:38.126552 17184 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.126032 17193 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.127696 16916 raft_consensus.cc:2393] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate 71a10d4adec64d58ba645d68425ed531 in this term.
I20260709 13:05:38.127542 17105 leader_election.cc:304] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:38.116778 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.128787 16991 raft_consensus.cc:2468] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:38.129623 17211 raft_consensus.cc:2804] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.129832 17105 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.130146 17211 raft_consensus.cc:493] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.130568 17211 raft_consensus.cc:3060] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.131199 17246 raft_consensus.cc:3060] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.133939 17184 tablet_bootstrap.cc:654] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.134152 17183 tablet_bootstrap.cc:492] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:38.140534 17246 raft_consensus.cc:2749] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.144302 17211 raft_consensus.cc:515] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.144819 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:38.145853 17227 raft_consensus.cc:515] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.148317 17227 leader_election.cc:290] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.149415 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
W20260709 13:05:38.151484 17031 leader_election.cc:343] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.153090 17183 tablet_bootstrap.cc:654] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.154942 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:38.155660 16991 raft_consensus.cc:3060] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.159075 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:38.159825 16916 raft_consensus.cc:3060] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.161136 17211 leader_election.cc:290] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.162736 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:38.163703 16989 raft_consensus.cc:3060] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.165222 17193 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.170857 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:38.171033 16991 raft_consensus.cc:2468] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:38.173806 17255 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.173538 17192 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.174351 17030 leader_election.cc:304] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:38.175712 17192 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.174305 17255 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.177912 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:38.180006 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:38.180156 17255 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.180751 17138 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:38.181021 16954 leader_election.cc:343] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.182720 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:38.183525 17137 raft_consensus.cc:2468] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:38.188665 17185 ts_tablet_manager.cc:1434] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.156s	user 0.009s	sys 0.000s
I20260709 13:05:38.171994 17211 raft_consensus.cc:493] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.190739 16954 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:38.199949 17211 raft_consensus.cc:515] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.200547 17255 raft_consensus.cc:2804] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.201036 17255 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.201480 17255 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.202483 17185 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:38.190757 16879 leader_election.cc:343] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.203378 16990 raft_consensus.cc:2468] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:38.203637 17251 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.204191 17251 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.204977 17030 leader_election.cc:304] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:38.206142 17198 raft_consensus.cc:2804] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.206715 17198 raft_consensus.cc:697] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:38.185954 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.190964 16989 raft_consensus.cc:2468] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:38.207572 17198 consensus_queue.cc:237] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.208424 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.209383 17251 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:38.210290 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.210458 17104 leader_election.cc:304] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:38.211855 16991 raft_consensus.cc:2393] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
W20260709 13:05:38.212399 16881 leader_election.cc:343] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.193279 17139 raft_consensus.cc:2468] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:38.201116 17184 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.214017 17184 ts_tablet_manager.cc:1403] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.088s	user 0.016s	sys 0.006s
I20260709 13:05:38.214504 16879 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:38.215061 16954 leader_election.cc:343] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.215756 17251 raft_consensus.cc:2804] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.216609 17251 raft_consensus.cc:697] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:38.217049 17184 raft_consensus.cc:359] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.217509 17251 consensus_queue.cc:237] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.217976 17184 raft_consensus.cc:385] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.217808 17255 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.218393 17184 raft_consensus.cc:740] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.219123 17184 consensus_queue.cc:260] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.193862 17186 tablet_bootstrap.cc:492] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:38.220360 17186 ts_tablet_manager.cc:1403] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.187s	user 0.041s	sys 0.033s
I20260709 13:05:38.198678 17227 raft_consensus.cc:2804] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.221263 16991 raft_consensus.cc:3060] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.222059 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:38.222319 17227 raft_consensus.cc:697] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:38.223373 17227 consensus_queue.cc:237] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:38.223878 16954 leader_election.cc:343] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.221081 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.228390 17184 ts_tablet_manager.cc:1434] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:05:38.228972 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.226151 17138 raft_consensus.cc:2468] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:38.229490 17184 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.226272 17186 raft_consensus.cc:359] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.230273 17186 raft_consensus.cc:385] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.230898 17102 leader_election.cc:304] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.231459 17211 leader_election.cc:290] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:38.238762 17191 raft_consensus.cc:3060] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.239200 17186 raft_consensus.cc:740] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.240562 17186 consensus_queue.cc:260] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.239720 17255 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.244865 17230 consensus_queue.cc:1048] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:38.236919 17184 tablet_bootstrap.cc:654] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.236977 17185 tablet_bootstrap.cc:654] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:38.243649 16879 leader_election.cc:343] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.244433 16954 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.242784 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:38.242818 17186 ts_tablet_manager.cc:1434] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 13:05:38.252672 17186 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:38.221707 17246 raft_consensus.cc:2804] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.249650 17191 raft_consensus.cc:2749] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.257275 17255 raft_consensus.cc:2804] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.256733 17139 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.259177 17255 raft_consensus.cc:697] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:38.260114 17255 consensus_queue.cc:237] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.255306 16879 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.261750 17246 raft_consensus.cc:697] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:38.262892 17251 raft_consensus.cc:2749] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.262706 17246 consensus_queue.cc:237] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.271719 17183 tablet_bootstrap.cc:492] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:38.272295 17183 ts_tablet_manager.cc:1403] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.138s	user 0.015s	sys 0.037s
I20260709 13:05:38.279110 16916 raft_consensus.cc:2468] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:38.280678 17191 raft_consensus.cc:493] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.284396 17211 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.286624 17183 raft_consensus.cc:359] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.287575 17183 raft_consensus.cc:385] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.287901 17183 raft_consensus.cc:740] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.288695 17183 consensus_queue.cc:260] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.286161 17211 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.292450 17211 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.292992 17211 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.293669 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.293895 17191 raft_consensus.cc:515] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.284651 17139 raft_consensus.cc:2468] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:38.295908 16954 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.296181 17191 leader_election.cc:290] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.297246 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.297600 17255 raft_consensus.cc:2804] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.298126 16915 raft_consensus.cc:2468] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
W20260709 13:05:38.298287 17104 leader_election.cc:343] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.299335 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.298386 17255 raft_consensus.cc:697] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:38.300707 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.301426 16916 raft_consensus.cc:2468] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:38.299986 16990 raft_consensus.cc:2468] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:38.302291 17105 leader_election.cc:304] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:38.300968 17255 consensus_queue.cc:237] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.293473 17211 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.305722 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.306414 17064 raft_consensus.cc:2468] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:38.293263 16814 catalog_manager.cc:5697] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:38.307927 17102 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:38.308534 17186 tablet_bootstrap.cc:654] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.309449 17211 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.310165 17211 raft_consensus.cc:2804] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.310509 17211 raft_consensus.cc:493] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.310861 17211 raft_consensus.cc:3060] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.312323 16813 catalog_manager.cc:5697] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:38.317544 17105 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:38.324766 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.315083 17246 raft_consensus.cc:2804] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.326045 17246 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.326426 17246 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:38.334828 17105 leader_election.cc:343] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.315384 17198 consensus_queue.cc:1048] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 13:05:38.340795 17191 raft_consensus.cc:2804] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.356098 17191 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.365751 17191 raft_consensus.cc:3060] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.364248 16812 catalog_manager.cc:5697] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:38.368765 16812 catalog_manager.cc:5697] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:38.380357 17246 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.381579 17183 ts_tablet_manager.cc:1434] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.109s	user 0.017s	sys 0.010s
I20260709 13:05:38.395874 17191 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.401182 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:38.401412 17191 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.401889 16916 raft_consensus.cc:3060] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.402129 17191 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.402745 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
W20260709 13:05:38.404373 17104 leader_election.cc:343] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.406394 17183 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:38.414292 17184 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.414350 17185 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:38.415104 17184 ts_tablet_manager.cc:1403] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.186s	user 0.015s	sys 0.024s
I20260709 13:05:38.415136 17185 ts_tablet_manager.cc:1403] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.213s	user 0.008s	sys 0.011s
I20260709 13:05:38.417652 17211 raft_consensus.cc:515] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.422124 16916 raft_consensus.cc:2468] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:38.422858 17211 leader_election.cc:290] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:38.420342 17184 raft_consensus.cc:359] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.425239 17184 raft_consensus.cc:385] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.425596 17184 raft_consensus.cc:740] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.403136 17191 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.426400 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:38.426321 17184 consensus_queue.cc:260] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.427026 16991 raft_consensus.cc:3060] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.419768 17186 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:38.427752 17186 ts_tablet_manager.cc:1403] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.175s	user 0.014s	sys 0.011s
I20260709 13:05:38.428161 17105 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:38.430575 17186 raft_consensus.cc:359] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.434042 17186 raft_consensus.cc:385] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.434309 17186 raft_consensus.cc:740] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.435940 17186 consensus_queue.cc:260] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.418267 17185 raft_consensus.cc:359] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.437098 17185 raft_consensus.cc:385] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.437294 17185 raft_consensus.cc:740] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.438539 17185 consensus_queue.cc:260] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.439870 17186 ts_tablet_manager.cc:1434] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:05:38.440869 17186 tablet_bootstrap.cc:492] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:38.444492 16991 raft_consensus.cc:2468] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:38.445806 17186 tablet_bootstrap.cc:654] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.435366 17211 raft_consensus.cc:2804] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.446110 17104 leader_election.cc:304] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:38.446372 17211 raft_consensus.cc:697] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:38.447719 17263 raft_consensus.cc:2804] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.447381 17211 consensus_queue.cc:237] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.448432 17263 raft_consensus.cc:697] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:38.432832 16813 catalog_manager.cc:5697] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:38.449659 17263 consensus_queue.cc:237] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.452692 17185 ts_tablet_manager.cc:1434] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.000s
I20260709 13:05:38.453766 17191 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:38.454949 17185 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:38.457199 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:38.457861 16916 raft_consensus.cc:3060] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.461881 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:38.463523 17102 leader_election.cc:343] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.465173 17183 tablet_bootstrap.cc:654] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.471436 17227 raft_consensus.cc:493] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.472059 17227 raft_consensus.cc:515] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.472553 17255 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.473080 17255 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.474545 17227 leader_election.cc:290] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.475284 17255 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.475788 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.475986 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:38.476667 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.482285 17184 ts_tablet_manager.cc:1434] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.067s	user 0.006s	sys 0.001s
I20260709 13:05:38.483894 17246 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.484786 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:38.485435 17064 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.488291 16916 raft_consensus.cc:2468] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
W20260709 13:05:38.477807 17031 leader_election.cc:343] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.477334 16990 raft_consensus.cc:2468] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:38.489065 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.489929 16915 raft_consensus.cc:2393] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:38.490162 17030 leader_election.cc:304] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:38.491235 16957 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:38.492234 17184 tablet_bootstrap.cc:492] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.494020 17255 raft_consensus.cc:3060] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.497700 17185 tablet_bootstrap.cc:654] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.493076 17227 raft_consensus.cc:2804] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.504369 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.502871 17227 raft_consensus.cc:493] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.505678 17227 raft_consensus.cc:3060] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.505968 17186 tablet_bootstrap.cc:492] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:38.506455 17186 ts_tablet_manager.cc:1403] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.066s	user 0.012s	sys 0.003s
I20260709 13:05:38.507718 16811 catalog_manager.cc:5697] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:38.509300 17186 raft_consensus.cc:359] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.510102 17186 raft_consensus.cc:385] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.510370 17186 raft_consensus.cc:740] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.518276 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:38.519368 16915 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.531611 17184 tablet_bootstrap.cc:654] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.532862 17227 raft_consensus.cc:515] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.535959 17198 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.536104 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:38.536692 17227 leader_election.cc:290] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.536772 16990 raft_consensus.cc:3060] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.536602 17198 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.539023 17198 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:38.540112 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:38.540449 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:38.541669 17031 leader_election.cc:343] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.546350 16915 raft_consensus.cc:2468] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:38.547390 16990 raft_consensus.cc:2468] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:38.546628 16812 catalog_manager.cc:5697] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:38.528872 17186 consensus_queue.cc:260] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.552275 17030 leader_election.cc:304] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:38.550216 17255 raft_consensus.cc:2749] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.550216 17064 raft_consensus.cc:2468] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
W20260709 13:05:38.552230 17104 leader_election.cc:343] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.557356 17104 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.562476 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:38.563390 17139 raft_consensus.cc:2393] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:38.563462 17263 raft_consensus.cc:2749] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.564522 17105 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:38.565354 17263 raft_consensus.cc:2804] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.565932 17263 raft_consensus.cc:697] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:38.566820 17263 consensus_queue.cc:237] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.567605 17028 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:38.570804 17183 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:38.571349 17183 ts_tablet_manager.cc:1403] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.174s	user 0.018s	sys 0.012s
I20260709 13:05:38.573689 17227 raft_consensus.cc:2804] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.574680 17230 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.574636 17227 raft_consensus.cc:697] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:38.576326 17183 raft_consensus.cc:359] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.578888 17183 raft_consensus.cc:385] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.579300 17183 raft_consensus.cc:740] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.581732 17183 consensus_queue.cc:260] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.582468 17186 ts_tablet_manager.cc:1434] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.076s	user 0.008s	sys 0.000s
I20260709 13:05:38.583670 17186 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:38.590659 17227 consensus_queue.cc:237] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.593031 17230 raft_consensus.cc:2749] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.595096 17185 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:38.595737 17185 ts_tablet_manager.cc:1403] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.141s	user 0.010s	sys 0.010s
I20260709 13:05:38.600541 17183 ts_tablet_manager.cc:1434] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.002s
I20260709 13:05:38.602291 17183 tablet_bootstrap.cc:492] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:38.603538 17185 raft_consensus.cc:359] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.604363 17185 raft_consensus.cc:385] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.604655 17185 raft_consensus.cc:740] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.605512 17185 consensus_queue.cc:260] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.608271 17185 ts_tablet_manager.cc:1434] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:05:38.609551 17185 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:38.615103 16879 consensus_peers.cc:597] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:38.616395 17185 tablet_bootstrap.cc:654] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.617559 17184 tablet_bootstrap.cc:492] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.618103 17184 ts_tablet_manager.cc:1403] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.135s	user 0.013s	sys 0.029s
I20260709 13:05:38.620234 17186 tablet_bootstrap.cc:654] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.621121 17184 raft_consensus.cc:359] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.612862 16812 catalog_manager.cc:5697] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:38.622198 17184 raft_consensus.cc:385] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.622702 17184 raft_consensus.cc:740] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.640023 17184 consensus_queue.cc:260] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.642311 17184 ts_tablet_manager.cc:1434] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 13:05:38.643286 17184 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.623260 16812 catalog_manager.cc:5697] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:38.645876 17186 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:38.646404 17186 ts_tablet_manager.cc:1403] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.063s	user 0.014s	sys 0.001s
W20260709 13:05:38.633215 17031 consensus_peers.cc:597] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:38.649441 17186 raft_consensus.cc:359] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.650420 17186 raft_consensus.cc:385] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.650713 17186 raft_consensus.cc:740] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.651656 17184 tablet_bootstrap.cc:654] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.651518 17186 consensus_queue.cc:260] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.644181 16813 catalog_manager.cc:5697] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:38.655005 17185 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:38.635521 17183 tablet_bootstrap.cc:654] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.655606 17185 ts_tablet_manager.cc:1403] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.046s	user 0.009s	sys 0.008s
I20260709 13:05:38.661167 17251 consensus_queue.cc:1048] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:38.664324 17184 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.664903 17184 ts_tablet_manager.cc:1403] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.003s
I20260709 13:05:38.658669 17185 raft_consensus.cc:359] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.667102 17185 raft_consensus.cc:385] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.667562 17185 raft_consensus.cc:740] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.667810 17184 raft_consensus.cc:359] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.668545 17184 raft_consensus.cc:385] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.668876 17184 raft_consensus.cc:740] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.669848 17184 consensus_queue.cc:260] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.668498 17185 consensus_queue.cc:260] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.672343 17184 ts_tablet_manager.cc:1434] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.006s
I20260709 13:05:38.672554 17186 ts_tablet_manager.cc:1434] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260709 13:05:38.673659 17184 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.676390 17185 ts_tablet_manager.cc:1434] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.020s	user 0.003s	sys 0.006s
I20260709 13:05:38.677451 17185 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:38.680645 17184 tablet_bootstrap.cc:654] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.673682 17186 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:38.706887 17185 tablet_bootstrap.cc:654] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.717255 17064 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.719878 17186 tablet_bootstrap.cc:654] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.722518 17227 consensus_queue.cc:1048] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:38.723939 17211 raft_consensus.cc:493] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.724524 17211 raft_consensus.cc:515] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.727097 17211 leader_election.cc:290] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.728236 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.730443 17065 raft_consensus.cc:2468] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:38.732331 17184 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.732859 17184 ts_tablet_manager.cc:1403] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.059s	user 0.009s	sys 0.032s
I20260709 13:05:38.734426 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:38.737540 17255 consensus_queue.cc:1048] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:38.740821 17102 leader_election.cc:304] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
W20260709 13:05:38.742276 17105 leader_election.cc:343] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.744045 17211 raft_consensus.cc:2804] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.744647 17211 raft_consensus.cc:493] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.735759 17184 raft_consensus.cc:359] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.749588 17184 raft_consensus.cc:385] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.750036 17184 raft_consensus.cc:740] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.750962 17184 consensus_queue.cc:260] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.751530 17230 consensus_queue.cc:1048] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:38.753469 17184 ts_tablet_manager.cc:1434] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.002s
I20260709 13:05:38.754446 17184 tablet_bootstrap.cc:492] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.763213 17184 tablet_bootstrap.cc:654] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.745105 17211 raft_consensus.cc:3060] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.802192 17255 consensus_queue.cc:1048] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:38.806200 17198 consensus_queue.cc:1048] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:38.810685 17186 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:38.811472 17186 ts_tablet_manager.cc:1403] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.138s	user 0.009s	sys 0.003s
I20260709 13:05:38.813844 17183 tablet_bootstrap.cc:492] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:38.814653 17183 ts_tablet_manager.cc:1403] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.213s	user 0.008s	sys 0.022s
I20260709 13:05:38.824590 17211 raft_consensus.cc:515] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:38.828673 17186 raft_consensus.cc:359] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.829732 17186 raft_consensus.cc:385] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.830083 17186 raft_consensus.cc:740] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.834218 17185 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:38.834889 17185 ts_tablet_manager.cc:1403] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.158s	user 0.053s	sys 0.054s
I20260709 13:05:38.836076 17184 tablet_bootstrap.cc:492] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:38.836606 17184 ts_tablet_manager.cc:1403] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.082s	user 0.012s	sys 0.006s
I20260709 13:05:38.839093 17183 raft_consensus.cc:359] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.839893 17183 raft_consensus.cc:385] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.840129 17185 raft_consensus.cc:359] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.840821 17185 raft_consensus.cc:385] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.841080 17185 raft_consensus.cc:740] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.842343 17295 consensus_queue.cc:1048] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:38.842605 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:38.843218 17064 raft_consensus.cc:3060] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.842985 17184 raft_consensus.cc:359] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.843822 17184 raft_consensus.cc:385] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.841991 17185 consensus_queue.cc:260] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.844137 17184 raft_consensus.cc:740] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.847048 17251 raft_consensus.cc:493] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.847200 17184 consensus_queue.cc:260] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.847604 17251 raft_consensus.cc:515] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.840180 17183 raft_consensus.cc:740] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.851266 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.851756 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:38.852447 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.853284 16991 raft_consensus.cc:2393] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:38.853140 17251 leader_election.cc:290] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
W20260709 13:05:38.853176 17105 leader_election.cc:343] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.853591 17183 consensus_queue.cc:260] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.853281 17211 leader_election.cc:290] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:38.855113 17185 ts_tablet_manager.cc:1434] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.002s
I20260709 13:05:38.854926 16881 leader_election.cc:304] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.855437 17186 consensus_queue.cc:260] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.856094 17185 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:38.857002 17272 consensus_queue.cc:1048] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:38.858402 17272 raft_consensus.cc:493] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.858824 17272 raft_consensus.cc:515] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.859789 17251 raft_consensus.cc:3060] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.861152 17295 consensus_queue.cc:1048] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:38.863864 17272 leader_election.cc:290] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:38.865617 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.866309 16991 raft_consensus.cc:2468] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:38.866803 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.867126 17183 ts_tablet_manager.cc:1434] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.052s	user 0.008s	sys 0.000s
I20260709 13:05:38.867682 17104 leader_election.cc:304] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:38.867835 17272 consensus_queue.cc:1048] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:38.868654 17263 raft_consensus.cc:2804] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:38.869022 17263 raft_consensus.cc:493] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:38.869445 17263 raft_consensus.cc:3060] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.870172 17186 ts_tablet_manager.cc:1434] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.058s	user 0.010s	sys 0.000s
W20260709 13:05:38.870255 17102 leader_election.cc:343] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.871285 17186 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:38.876646 17064 raft_consensus.cc:2468] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:38.868130 17183 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:38.878176 17102 leader_election.cc:304] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:38.883294 17185 tablet_bootstrap.cc:654] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.890400 17271 raft_consensus.cc:2804] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:38.892644 17271 raft_consensus.cc:697] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:38.894057 17271 consensus_queue.cc:237] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:38.897068 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:38.910274 17193 raft_consensus.cc:493] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.910454 17246 consensus_queue.cc:1048] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:38.923192 17193 raft_consensus.cc:515] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.932868 17184 ts_tablet_manager.cc:1434] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.096s	user 0.009s	sys 0.000s
I20260709 13:05:38.933952 17184 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:38.935721 17251 raft_consensus.cc:2749] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:38.938409 17186 tablet_bootstrap.cc:654] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.945358 17263 raft_consensus.cc:515] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:38.947243 17183 tablet_bootstrap.cc:654] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.947744 17263 leader_election.cc:290] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:38.948658 17263 consensus_queue.cc:1048] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:38.949477 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:38.949854 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:38.950285 16991 raft_consensus.cc:3060] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:38.921346 17272 consensus_queue.cc:1048] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260709 13:05:38.929481 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:38.931125 17193 leader_election.cc:290] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:38.963618 17184 tablet_bootstrap.cc:654] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:38.965996 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
W20260709 13:05:38.968250 17102 leader_election.cc:343] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:38.979094 17185 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:38.979573 17185 ts_tablet_manager.cc:1403] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.124s	user 0.014s	sys 0.010s
I20260709 13:05:38.955636 17251 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:38.982946 17251 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:38.990808 17185 raft_consensus.cc:359] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.994234 17185 raft_consensus.cc:385] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:38.994491 17185 raft_consensus.cc:740] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:38.995097 17185 consensus_queue.cc:260] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:38.999819 16881 leader_election.cc:304] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:38.997233 17185 ts_tablet_manager.cc:1434] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.002s
I20260709 13:05:39.003913 17185 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:39.004915 17255 raft_consensus.cc:493] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.003432 17193 raft_consensus.cc:2749] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:39.003393 17251 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:39.011708 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:39.015750 17255 raft_consensus.cc:515] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.021504 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:39.022253 17065 raft_consensus.cc:2468] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:39.024101 16879 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:39.025364 16990 raft_consensus.cc:2468] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:39.026060 17255 leader_election.cc:290] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:39.028505 17191 consensus_queue.cc:1048] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:39.028827 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:39.029593 17137 raft_consensus.cc:2468] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:39.030807 16912 raft_consensus.cc:3060] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.031023 16954 leader_election.cc:304] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:39.031827 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:39.033092 17255 raft_consensus.cc:2804] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.033561 17255 raft_consensus.cc:493] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.033900 17255 raft_consensus.cc:3060] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:39.034523 16957 leader_election.cc:343] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:39.043100 17193 raft_consensus.cc:2804] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.043699 17193 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.044139 17193 raft_consensus.cc:3060] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.046394 16991 raft_consensus.cc:2468] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:39.047889 17104 leader_election.cc:304] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:39.048908 17185 tablet_bootstrap.cc:654] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.050160 17191 raft_consensus.cc:2804] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.051090 17191 raft_consensus.cc:697] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:39.052788 17191 consensus_queue.cc:237] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.054347 17183 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:39.055095 17183 ts_tablet_manager.cc:1403] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.187s	user 0.025s	sys 0.023s
I20260709 13:05:39.060153 17230 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.061002 17230 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.064554 17230 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:39.064980 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:39.060402 17186 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:39.067317 17186 ts_tablet_manager.cc:1403] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.196s	user 0.011s	sys 0.026s
I20260709 13:05:39.092252 17186 raft_consensus.cc:359] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.094748 17193 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.095918 17198 consensus_queue.cc:1048] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.093283 17186 raft_consensus.cc:385] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.097800 17186 raft_consensus.cc:740] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.096290 17255 raft_consensus.cc:515] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.081212 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:39.103724 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:39.104456 16991 raft_consensus.cc:3060] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.107790 17031 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:39.100127 17186 consensus_queue.cc:260] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.109182 17198 raft_consensus.cc:2749] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:39.110786 17255 leader_election.cc:290] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:39.111990 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:39.114400 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:39.114758 16957 leader_election.cc:343] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:39.115141 17139 raft_consensus.cc:3060] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.117525 17183 raft_consensus.cc:359] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.120296 17183 raft_consensus.cc:385] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.120601 17183 raft_consensus.cc:740] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.121598 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:39.122316 17065 raft_consensus.cc:3060] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.127369 17186 ts_tablet_manager.cc:1434] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.060s	user 0.007s	sys 0.003s
I20260709 13:05:39.119843 17193 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:39.130656 17183 consensus_queue.cc:260] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.178313 17186 tablet_bootstrap.cc:492] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:39.179082 17183 ts_tablet_manager.cc:1434] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.124s	user 0.005s	sys 0.003s
I20260709 13:05:39.180142 17183 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:39.181586 17255 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.182107 17255 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.133183 17139 raft_consensus.cc:2468] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:39.147503 17198 consensus_queue.cc:1048] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.184347 17255 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:39.155200 17263 consensus_queue.cc:1048] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:39.186266 16954 leader_election.cc:304] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:39.186765 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:39.160522 17064 raft_consensus.cc:3060] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.163232 17185 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:39.175351 17184 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:39.193933 17185 ts_tablet_manager.cc:1403] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.191s	user 0.041s	sys 0.013s
I20260709 13:05:39.194324 17184 ts_tablet_manager.cc:1403] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.261s	user 0.019s	sys 0.012s
I20260709 13:05:39.196785 17185 raft_consensus.cc:359] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.198066 17185 raft_consensus.cc:385] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.199110 17185 raft_consensus.cc:740] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.199921 17185 consensus_queue.cc:260] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.198560 17272 consensus_queue.cc:1048] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.200706 17184 raft_consensus.cc:359] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.203874 17184 raft_consensus.cc:385] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.204195 17184 raft_consensus.cc:740] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
W20260709 13:05:39.201681 16954 leader_election.cc:343] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:39.205302 17184 consensus_queue.cc:260] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.207949 17184 ts_tablet_manager.cc:1434] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20260709 13:05:39.225143 17184 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:39.201830 17255 raft_consensus.cc:2804] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.227164 17255 raft_consensus.cc:697] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:39.228312 17255 consensus_queue.cc:237] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.240141 17295 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.240759 17295 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.245635 17295 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:39.244671 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:39.247771 16914 raft_consensus.cc:2468] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:39.248075 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:39.249213 16957 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:39.185498 17186 tablet_bootstrap.cc:654] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.203225 17183 tablet_bootstrap.cc:654] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.205734 17185 ts_tablet_manager.cc:1434] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 13:05:39.213265 17198 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.137048 17065 raft_consensus.cc:2468] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:39.251833 17198 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.254541 16879 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:39.255564 17193 raft_consensus.cc:2804] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.256068 17193 raft_consensus.cc:697] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:39.256208 17064 raft_consensus.cc:2468] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:39.257002 17193 consensus_queue.cc:237] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.259553 17185 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:39.217128 16812 catalog_manager.cc:5697] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:39.182308 16991 raft_consensus.cc:2468] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:39.250257 17295 raft_consensus.cc:2804] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.202576 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:39.262665 17295 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.263321 17295 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.270941 17183 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:39.271485 17183 ts_tablet_manager.cc:1403] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.092s	user 0.007s	sys 0.007s
I20260709 13:05:39.275131 17185 tablet_bootstrap.cc:654] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.275625 17198 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:39.274902 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:39.277556 16914 raft_consensus.cc:3060] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.278668 16912 raft_consensus.cc:2468] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:39.278234 17139 raft_consensus.cc:2468] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:39.280057 17031 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:39.280963 17198 raft_consensus.cc:2804] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.281358 17198 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.263928 16812 catalog_manager.cc:5697] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:39.281766 17198 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.294660 16954 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:39.295265 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:39.294984 17295 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.303503 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:39.304164 16912 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.304836 17295 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:39.305523 16991 raft_consensus.cc:2393] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:39.305480 17295 raft_consensus.cc:2804] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.308969 17295 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.309520 17272 consensus_queue.cc:1048] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:05:39.315443 17198 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.312842 16912 raft_consensus.cc:2468] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:39.309387 17295 raft_consensus.cc:3060] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.342689 16811 catalog_manager.cc:5697] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:39.346410 16957 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:39.325127 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:39.326606 17185 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:39.347381 17185 ts_tablet_manager.cc:1403] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.094s	user 0.012s	sys 0.012s
I20260709 13:05:39.348925 17328 raft_consensus.cc:2804] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.316812 17183 raft_consensus.cc:359] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.346236 17272 consensus_queue.cc:1048] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.350574 17183 raft_consensus.cc:385] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.350322 17185 raft_consensus.cc:359] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.350976 17183 raft_consensus.cc:740] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.351182 17185 raft_consensus.cc:385] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.351509 17185 raft_consensus.cc:740] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.351753 17183 consensus_queue.cc:260] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.352275 17185 consensus_queue.cc:260] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.355202 17185 ts_tablet_manager.cc:1434] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:05:39.355301 17064 raft_consensus.cc:2393] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:39.356492 17185 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:39.356812 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:39.357897 16912 raft_consensus.cc:2393] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:39.359755 17198 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:39.360945 17183 ts_tablet_manager.cc:1434] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.089s	user 0.009s	sys 0.000s
I20260709 13:05:39.362010 17183 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:39.362941 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:39.350375 17295 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.366010 17185 tablet_bootstrap.cc:654] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.331725 17184 tablet_bootstrap.cc:654] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.370493 17186 tablet_bootstrap.cc:492] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:39.371156 17186 ts_tablet_manager.cc:1403] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.193s	user 0.004s	sys 0.012s
I20260709 13:05:39.369846 17328 raft_consensus.cc:697] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:39.373035 17328 consensus_queue.cc:237] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.376681 17295 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:39.380599 17193 raft_consensus.cc:493] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.381362 17193 raft_consensus.cc:515] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.387130 17193 leader_election.cc:290] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:39.388880 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:39.389891 17137 raft_consensus.cc:3060] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.387068 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:39.404241 17183 tablet_bootstrap.cc:654] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.410990 17137 raft_consensus.cc:2468] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:39.405103 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:39.410008 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:39.419100 17139 raft_consensus.cc:2468] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
W20260709 13:05:39.420368 16954 leader_election.cc:343] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:39.421092 17030 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:39.421196 16954 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:39.422293 16879 leader_election.cc:304] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:39.423194 17193 raft_consensus.cc:2804] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.422665 17198 raft_consensus.cc:2749] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:05:39.423595 17193 raft_consensus.cc:493] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.424232 17193 raft_consensus.cc:3060] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.423530 17324 raft_consensus.cc:2804] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:05:39.423619 16881 leader_election.cc:343] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:39.425071 17324 raft_consensus.cc:697] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:39.426177 17324 consensus_queue.cc:237] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.451294 17193 raft_consensus.cc:515] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.453540 17186 raft_consensus.cc:359] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.454748 17186 raft_consensus.cc:385] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.457701 17185 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:39.475906 17272 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.476219 17185 ts_tablet_manager.cc:1403] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.120s	user 0.026s	sys 0.016s
I20260709 13:05:39.476459 17272 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.478659 17272 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:39.467569 17184 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:39.479249 17185 raft_consensus.cc:359] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.479919 17184 ts_tablet_manager.cc:1403] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.255s	user 0.012s	sys 0.016s
I20260709 13:05:39.480000 17185 raft_consensus.cc:385] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.480386 17185 raft_consensus.cc:740] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.458089 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:39.455107 17186 raft_consensus.cc:740] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.464196 17183 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:39.482925 17183 ts_tablet_manager.cc:1403] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.121s	user 0.009s	sys 0.020s
I20260709 13:05:39.456750 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:39.486423 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:39.480078 17193 leader_election.cc:290] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:39.492545 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:39.501682 17065 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:39.494468 17184 raft_consensus.cc:359] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.503144 17184 raft_consensus.cc:385] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.503466 17184 raft_consensus.cc:740] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.503907 17102 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:39.504391 17184 consensus_queue.cc:260] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.507054 17184 ts_tablet_manager.cc:1434] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.001s
I20260709 13:05:39.507503 17272 raft_consensus.cc:2804] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.507910 17272 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.508177 17184 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:39.508244 17272 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.511274 16811 catalog_manager.cc:5697] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:39.513561 16989 raft_consensus.cc:3060] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.515162 17184 tablet_bootstrap.cc:654] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.485136 17186 consensus_queue.cc:260] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.497922 17185 consensus_queue.cc:260] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.519408 16814 catalog_manager.cc:5697] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:39.522897 17186 ts_tablet_manager.cc:1434] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.151s	user 0.009s	sys 0.000s
I20260709 13:05:39.523720 17186 tablet_bootstrap.cc:492] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:39.484653 17139 raft_consensus.cc:3060] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.521591 16912 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:39.529047 17185 ts_tablet_manager.cc:1434] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.052s	user 0.007s	sys 0.001s
I20260709 13:05:39.521687 16811 catalog_manager.cc:5697] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:39.530253 17185 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:39.533432 17183 raft_consensus.cc:359] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.534133 17183 raft_consensus.cc:385] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.534410 17183 raft_consensus.cc:740] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.536832 17185 tablet_bootstrap.cc:654] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.539734 17183 consensus_queue.cc:260] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.545495 17183 ts_tablet_manager.cc:1434] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.062s	user 0.007s	sys 0.000s
I20260709 13:05:39.539238 16989 raft_consensus.cc:2468] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:39.524796 17272 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.548707 17183 tablet_bootstrap.cc:492] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:39.549937 16881 leader_election.cc:304] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:39.550310 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:39.550879 17193 raft_consensus.cc:2804] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.550979 17065 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.551456 17193 raft_consensus.cc:697] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:39.552376 17193 consensus_queue.cc:237] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.556915 17186 tablet_bootstrap.cc:654] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.559935 17272 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:39.562006 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:39.562693 16912 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.566984 17183 tablet_bootstrap.cc:654] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.570317 17064 raft_consensus.cc:3060] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.575107 17065 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:39.576498 17102 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:39.577064 17139 raft_consensus.cc:2468] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:39.577368 17272 raft_consensus.cc:2804] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.579166 17272 raft_consensus.cc:697] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:39.586580 17191 consensus_queue.cc:1048] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.589531 17272 consensus_queue.cc:237] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.586841 16912 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:39.592232 17184 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:39.592993 17184 ts_tablet_manager.cc:1403] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.085s	user 0.012s	sys 0.003s
I20260709 13:05:39.597525 17186 tablet_bootstrap.cc:492] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:39.598664 17186 ts_tablet_manager.cc:1403] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.075s	user 0.013s	sys 0.005s
I20260709 13:05:39.601202 17183 tablet_bootstrap.cc:492] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:39.601353 17186 raft_consensus.cc:359] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.601770 17183 ts_tablet_manager.cc:1403] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.053s	user 0.006s	sys 0.011s
I20260709 13:05:39.602144 17186 raft_consensus.cc:385] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.602458 17186 raft_consensus.cc:740] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.604857 17183 raft_consensus.cc:359] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.605646 17183 raft_consensus.cc:385] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.605916 17183 raft_consensus.cc:740] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.606736 17183 consensus_queue.cc:260] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.609189 17183 ts_tablet_manager.cc:1434] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.002s
I20260709 13:05:39.610339 17183 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:39.610807 17184 raft_consensus.cc:359] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.611559 17184 raft_consensus.cc:385] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.611822 17184 raft_consensus.cc:740] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.613692 17184 consensus_queue.cc:260] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.603135 17186 consensus_queue.cc:260] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.617626 17186 ts_tablet_manager.cc:1434] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 13:05:39.617930 17185 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:39.618503 17185 ts_tablet_manager.cc:1403] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.089s	user 0.002s	sys 0.014s
I20260709 13:05:39.618637 17186 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:39.619889 17183 tablet_bootstrap.cc:654] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.621637 17185 raft_consensus.cc:359] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.622359 17185 raft_consensus.cc:385] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.622625 17185 raft_consensus.cc:740] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.623602 17185 consensus_queue.cc:260] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.626649 16814 catalog_manager.cc:5697] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:39.626711 17185 ts_tablet_manager.cc:1434] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:05:39.629259 17185 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:39.631948 17184 ts_tablet_manager.cc:1434] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.039s	user 0.007s	sys 0.000s
I20260709 13:05:39.636956 17184 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:39.641790 17185 tablet_bootstrap.cc:654] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.646734 17272 consensus_queue.cc:1048] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.650823 17324 consensus_queue.cc:1048] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:05:39.654109 17186 tablet_bootstrap.cc:654] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.679673 17184 tablet_bootstrap.cc:654] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.678329 17193 raft_consensus.cc:493] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.690737 17193 raft_consensus.cc:515] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.678270 17324 raft_consensus.cc:493] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.694190 17324 raft_consensus.cc:515] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.698688 17324 leader_election.cc:290] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:39.699386 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:39.700392 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:39.701066 16912 raft_consensus.cc:2468] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:39.701292 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:39.704217 16954 leader_election.cc:343] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:39.706840 17183 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:39.707406 17183 ts_tablet_manager.cc:1403] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.097s	user 0.014s	sys 0.032s
I20260709 13:05:39.708127 16957 leader_election.cc:304] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:39.715807 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:39.715590 17183 raft_consensus.cc:359] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.716662 17183 raft_consensus.cc:385] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.716943 17183 raft_consensus.cc:740] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.714303 17193 leader_election.cc:290] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:39.717682 17183 consensus_queue.cc:260] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.718278 16881 leader_election.cc:304] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:39.716482 17324 raft_consensus.cc:2804] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.719305 17324 raft_consensus.cc:493] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.719688 17324 raft_consensus.cc:3060] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.719903 17183 ts_tablet_manager.cc:1434] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:05:39.720888 17183 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:39.720989 17185 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:39.721632 17185 ts_tablet_manager.cc:1403] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.093s	user 0.024s	sys 0.017s
I20260709 13:05:39.727494 16812 catalog_manager.cc:5697] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:39.740792 17304 raft_consensus.cc:2749] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:39.741866 17184 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:39.742419 17184 ts_tablet_manager.cc:1403] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.106s	user 0.014s	sys 0.008s
I20260709 13:05:39.742659 17185 raft_consensus.cc:359] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.743569 17185 raft_consensus.cc:385] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.745115 17185 raft_consensus.cc:740] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.756451 17184 raft_consensus.cc:359] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.757925 17184 raft_consensus.cc:385] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.750422 16912 raft_consensus.cc:3060] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.758620 17184 raft_consensus.cc:740] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.760149 17184 consensus_queue.cc:260] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.765100 17184 ts_tablet_manager.cc:1434] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.001s
I20260709 13:05:39.766811 17184 tablet_bootstrap.cc:492] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:39.747433 17185 consensus_queue.cc:260] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.796696 17324 raft_consensus.cc:515] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.804097 17185 ts_tablet_manager.cc:1434] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.082s	user 0.006s	sys 0.002s
I20260709 13:05:39.811575 17185 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:39.815495 17183 tablet_bootstrap.cc:654] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.816787 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:39.819029 16914 raft_consensus.cc:3060] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.819416 17324 leader_election.cc:290] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:39.826146 17186 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:39.826802 17186 ts_tablet_manager.cc:1403] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.208s	user 0.018s	sys 0.028s
I20260709 13:05:39.827257 17184 tablet_bootstrap.cc:654] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.822636 17272 raft_consensus.cc:493] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:39.833485 17272 raft_consensus.cc:515] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.837019 17272 leader_election.cc:290] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:39.823946 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:39.838857 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3fbc4f00185436cbce1d8e24e6d1e3a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:39.839558 16913 raft_consensus.cc:2468] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:39.837237 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3fbc4f00185436cbce1d8e24e6d1e3a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:39.843158 16990 raft_consensus.cc:2468] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:39.845047 17104 leader_election.cc:304] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:39.850028 17353 consensus_queue.cc:1048] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.851004 17186 raft_consensus.cc:359] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.851719 17186 raft_consensus.cc:385] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.851545 17353 consensus_queue.cc:1048] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.852015 17186 raft_consensus.cc:740] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.852365 17272 raft_consensus.cc:2804] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:39.852823 17272 raft_consensus.cc:493] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:39.853645 17272 raft_consensus.cc:3060] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:39.859685 16954 leader_election.cc:343] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:39.852727 17186 consensus_queue.cc:260] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:39.862941 17330 consensus_queue.cc:1048] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:05:39.864995 17330 consensus_queue.cc:1048] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:39.875547 17272 raft_consensus.cc:515] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.906893 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3fbc4f00185436cbce1d8e24e6d1e3a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:39.907635 16913 raft_consensus.cc:3060] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.908600 17184 tablet_bootstrap.cc:492] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:39.909113 17184 ts_tablet_manager.cc:1403] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.143s	user 0.005s	sys 0.008s
I20260709 13:05:39.911927 17184 raft_consensus.cc:359] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.912591 17184 raft_consensus.cc:385] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.912854 17184 raft_consensus.cc:740] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.913519 17184 consensus_queue.cc:260] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:39.915380 17184 ts_tablet_manager.cc:1434] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:05:39.916024 17184 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:39.878219 17186 ts_tablet_manager.cc:1434] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.051s	user 0.007s	sys 0.000s
I20260709 13:05:39.918345 17186 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:39.922237 16914 raft_consensus.cc:2468] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:39.924795 17185 tablet_bootstrap.cc:654] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.897857 17255 consensus_queue.cc:1048] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:39.902871 17183 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:39.923804 16957 leader_election.cc:304] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:39.929306 17183 ts_tablet_manager.cc:1403] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.209s	user 0.030s	sys 0.042s
I20260709 13:05:39.933315 17330 raft_consensus.cc:2804] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.934067 17330 raft_consensus.cc:697] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:39.935243 17330 consensus_queue.cc:237] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:39.951388 16913 raft_consensus.cc:2468] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:39.952694 17105 leader_election.cc:304] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:39.953734 17272 leader_election.cc:290] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:39.953933 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3fbc4f00185436cbce1d8e24e6d1e3a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:39.954480 17184 tablet_bootstrap.cc:654] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.955559 17186 tablet_bootstrap.cc:654] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:39.956048 16990 raft_consensus.cc:3060] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:39.953825 17263 raft_consensus.cc:2804] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:39.957787 17263 raft_consensus.cc:697] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:39.968535 17263 consensus_queue.cc:237] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.971841 17183 raft_consensus.cc:359] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.973500 17183 raft_consensus.cc:385] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:39.974121 17183 raft_consensus.cc:740] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:39.977279 17183 consensus_queue.cc:260] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:39.982609 17185 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:39.983181 17185 ts_tablet_manager.cc:1403] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.172s	user 0.009s	sys 0.004s
I20260709 13:05:39.980109 17183 ts_tablet_manager.cc:1434] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.000s
I20260709 13:05:39.985072 17183 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:40.001646 17065 raft_consensus.cc:3060] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.003855 17352 consensus_queue.cc:1048] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:40.010020 17246 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.010712 17246 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.013733 17246 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:40.015553 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:40.042570 16990 raft_consensus.cc:2468] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:40.016383 17064 raft_consensus.cc:2468] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:40.046352 17185 raft_consensus.cc:359] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.047214 17185 raft_consensus.cc:385] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.047508 17185 raft_consensus.cc:740] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.048252 17185 consensus_queue.cc:260] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:40.050681 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:40.055037 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.055863 17328 consensus_queue.cc:1048] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:40.059520 17352 raft_consensus.cc:493] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.054183 17185 ts_tablet_manager.cc:1434] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.071s	user 0.008s	sys 0.002s
I20260709 13:05:40.061381 17185 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:40.059062 17102 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.038709 17324 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.039026 17246 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.089655 17246 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:40.091935 17105 leader_election.cc:343] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.087420 17324 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.060135 17352 raft_consensus.cc:515] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.126731 17184 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:40.131634 17184 ts_tablet_manager.cc:1403] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.216s	user 0.025s	sys 0.005s
I20260709 13:05:40.133949 17185 tablet_bootstrap.cc:654] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.138034 17183 tablet_bootstrap.cc:654] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.104929 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:40.099579 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:40.115229 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.098690 17324 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:40.109241 17255 consensus_queue.cc:1048] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20260709 13:05:40.117899 17263 consensus_queue.cc:1048] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
W20260709 13:05:40.148216 16881 leader_election.cc:343] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.093146 17246 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:40.106328 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:40.104158 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:40.152030 17139 raft_consensus.cc:2468] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:40.107473 17227 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.103312 17352 leader_election.cc:290] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:40.100816 17271 raft_consensus.cc:2804] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.153342 16954 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:40.153880 17271 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:40.099889 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.154413 17255 raft_consensus.cc:2804] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.155118 16990 raft_consensus.cc:2468] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:40.156653 17104 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
W20260709 13:05:40.156924 16879 leader_election.cc:343] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.153167 17227 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.157478 16879 leader_election.cc:304] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:40.158771 17352 raft_consensus.cc:2749] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:40.154379 17271 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.159968 17211 raft_consensus.cc:2804] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.160486 17211 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:40.160884 17211 raft_consensus.cc:3060] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.159080 17353 consensus_queue.cc:1048] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:40.166839 16813 catalog_manager.cc:5697] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:40.165598 17255 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:05:40.168053 17102 leader_election.cc:343] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.180928 17255 raft_consensus.cc:3060] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.188417 17186 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.190060 17186 ts_tablet_manager.cc:1403] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.272s	user 0.048s	sys 0.043s
W20260709 13:05:40.197145 16954 leader_election.cc:343] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.252779 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.262696 17227 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:40.264070 17186 raft_consensus.cc:359] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.264998 17186 raft_consensus.cc:385] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.264748 17211 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.267303 17211 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:40.269904 17184 raft_consensus.cc:359] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.271843 17184 raft_consensus.cc:385] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.272166 17184 raft_consensus.cc:740] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.272995 17184 consensus_queue.cc:260] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.273803 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:40.276082 17184 ts_tablet_manager.cc:1434] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.144s	user 0.008s	sys 0.000s
I20260709 13:05:40.277285 17184 tablet_bootstrap.cc:492] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:40.271256 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:40.265323 17186 raft_consensus.cc:740] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.282168 17186 consensus_queue.cc:260] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.289188 17028 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:40.291081 17211 consensus_queue.cc:1048] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:40.294255 17227 raft_consensus.cc:2749] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:40.296461 17185 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:40.297487 17185 ts_tablet_manager.cc:1403] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.236s	user 0.024s	sys 0.022s
I20260709 13:05:40.298920 17186 ts_tablet_manager.cc:1434] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.108s	user 0.006s	sys 0.001s
I20260709 13:05:40.289880 17255 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.314493 16989 raft_consensus.cc:2393] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:40.319958 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:40.328804 17271 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.334982 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:40.343158 17186 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
W20260709 13:05:40.343583 17102 leader_election.cc:343] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.343875 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:40.344586 17062 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.345518 17104 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:40.342100 17030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 53629.5us
I20260709 13:05:40.349476 17375 raft_consensus.cc:2749] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:05:40.347285 17271 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:40.352079 17255 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:40.354511 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:40.358654 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:40.362160 17139 raft_consensus.cc:2393] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:40.362581 17186 tablet_bootstrap.cc:654] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.366567 17185 raft_consensus.cc:359] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.367403 17185 raft_consensus.cc:385] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.367772 17185 raft_consensus.cc:740] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.367944 17062 raft_consensus.cc:2468] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:40.368643 17185 consensus_queue.cc:260] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.369655 16914 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.370529 17185 ts_tablet_manager.cc:1434] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.073s	user 0.006s	sys 0.000s
I20260709 13:05:40.377676 17185 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:40.379853 16914 raft_consensus.cc:2468] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:40.382335 16814 catalog_manager.cc:5697] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:40.391024 16954 leader_election.cc:343] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.391661 16954 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:40.393009 17370 raft_consensus.cc:2749] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:05:40.395494 17271 raft_consensus.cc:493] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.396093 17271 raft_consensus.cc:515] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.397015 17102 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.399163 17271 leader_election.cc:290] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:40.401675 17271 raft_consensus.cc:2804] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:40.410727 17370 consensus_queue.cc:1048] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:40.413460 17271 raft_consensus.cc:697] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:40.408702 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:40.419291 17104 leader_election.cc:343] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.421865 17183 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:40.423408 17183 ts_tablet_manager.cc:1403] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.439s	user 0.029s	sys 0.033s
I20260709 13:05:40.427675 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:40.430282 17105 leader_election.cc:343] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.430887 17105 leader_election.cc:304] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:40.432039 17185 tablet_bootstrap.cc:654] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.432730 17184 tablet_bootstrap.cc:654] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.437286 17183 raft_consensus.cc:359] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.438128 17183 raft_consensus.cc:385] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.438423 17183 raft_consensus.cc:740] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.439199 17183 consensus_queue.cc:260] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.444612 17183 ts_tablet_manager.cc:1434] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 13:05:40.445922 17183 tablet_bootstrap.cc:492] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:40.450714 17186 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.451536 17186 ts_tablet_manager.cc:1403] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.109s	user 0.033s	sys 0.031s
I20260709 13:05:40.452769 17183 tablet_bootstrap.cc:654] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.462707 17186 raft_consensus.cc:359] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.463549 17186 raft_consensus.cc:385] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.463836 17186 raft_consensus.cc:740] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.466101 17271 consensus_queue.cc:237] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.432814 17375 raft_consensus.cc:2749] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:40.472007 17375 raft_consensus.cc:493] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.472571 17375 raft_consensus.cc:515] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.475106 17375 leader_election.cc:290] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:40.476390 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.477072 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.464536 17186 consensus_queue.cc:260] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:40.478763 17105 leader_election.cc:343] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.480587 17186 ts_tablet_manager.cc:1434] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.029s	user 0.004s	sys 0.003s
I20260709 13:05:40.481642 17186 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:40.495401 17064 raft_consensus.cc:3060] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.495723 17255 raft_consensus.cc:493] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.496662 17255 raft_consensus.cc:515] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.505389 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:40.507737 16954 leader_election.cc:343] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:40.488969 17104 leader_election.cc:343] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.515641 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.515877 17104 leader_election.cc:304] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:40.517481 17375 raft_consensus.cc:2749] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:05:40.517539 16957 leader_election.cc:343] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.518229 16957 leader_election.cc:304] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:40.519109 17324 raft_consensus.cc:2749] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:40.520936 17255 leader_election.cc:290] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:40.522673 17375 consensus_queue.cc:1048] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:40.523944 17263 consensus_queue.cc:1048] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:40.530717 17324 consensus_queue.cc:1048] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:40.539415 17186 tablet_bootstrap.cc:654] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.540247 17185 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:40.540822 17185 ts_tablet_manager.cc:1403] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.163s	user 0.017s	sys 0.004s
I20260709 13:05:40.540375 16811 catalog_manager.cc:5697] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:40.543953 17185 raft_consensus.cc:359] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.544646 17185 raft_consensus.cc:385] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.544889 17185 raft_consensus.cc:740] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.545598 17185 consensus_queue.cc:260] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.547925 17185 ts_tablet_manager.cc:1434] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:05:40.548898 17185 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:40.552659 17184 tablet_bootstrap.cc:492] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:40.556685 17184 ts_tablet_manager.cc:1403] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.280s	user 0.025s	sys 0.017s
I20260709 13:05:40.557425 17353 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.557947 17353 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.588423 17183 tablet_bootstrap.cc:492] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:40.589171 17183 ts_tablet_manager.cc:1403] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.143s	user 0.013s	sys 0.008s
I20260709 13:05:40.591323 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:40.592206 17183 raft_consensus.cc:359] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:40.592691 16879 leader_election.cc:343] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.593079 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:40.594733 16879 leader_election.cc:343] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.595201 16879 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:40.595997 17193 raft_consensus.cc:2749] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:40.592964 17183 raft_consensus.cc:385] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.596692 17183 raft_consensus.cc:740] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.602847 17353 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:40.597503 17183 consensus_queue.cc:260] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.608280 17183 ts_tablet_manager.cc:1434] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20260709 13:05:40.615825 17186 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.616425 17186 ts_tablet_manager.cc:1403] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.135s	user 0.010s	sys 0.013s
I20260709 13:05:40.616825 17279 raft_consensus.cc:493] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.617463 17279 raft_consensus.cc:515] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.619773 17279 leader_election.cc:290] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:40.619506 17186 raft_consensus.cc:359] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.620297 17186 raft_consensus.cc:385] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.620622 17186 raft_consensus.cc:740] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.621474 17183 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:40.622179 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.623299 16914 raft_consensus.cc:2468] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:40.625602 17031 leader_election.cc:304] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.626734 17279 raft_consensus.cc:2804] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.627178 17279 raft_consensus.cc:493] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:40.627543 17279 raft_consensus.cc:3060] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.628185 17183 tablet_bootstrap.cc:654] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.630326 17185 tablet_bootstrap.cc:654] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.629768 17186 consensus_queue.cc:260] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.622315 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.639106 17186 ts_tablet_manager.cc:1434] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
W20260709 13:05:40.641294 17030 leader_election.cc:343] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.642156 17186 tablet_bootstrap.cc:492] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:40.645483 17184 raft_consensus.cc:359] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.646241 17184 raft_consensus.cc:385] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.646517 17184 raft_consensus.cc:740] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.648422 17186 tablet_bootstrap.cc:654] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.651248 17279 raft_consensus.cc:515] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.659956 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:40.660691 16913 raft_consensus.cc:3060] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.652666 17184 consensus_queue.cc:260] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.666972 17183 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:40.667537 17183 ts_tablet_manager.cc:1403] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.058s	user 0.008s	sys 0.004s
I20260709 13:05:40.670245 16913 raft_consensus.cc:2468] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:40.670454 17183 raft_consensus.cc:359] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.672098 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:40.672492 17184 ts_tablet_manager.cc:1434] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.116s	user 0.003s	sys 0.003s
I20260709 13:05:40.677424 17184 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:40.678407 17186 tablet_bootstrap.cc:492] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.678957 17186 ts_tablet_manager.cc:1403] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.037s	user 0.009s	sys 0.003s
I20260709 13:05:40.672783 17279 leader_election.cc:290] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:40.681805 17186 raft_consensus.cc:359] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.682719 17186 raft_consensus.cc:385] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.683035 17186 raft_consensus.cc:740] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.683782 17186 consensus_queue.cc:260] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.673590 17183 raft_consensus.cc:385] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.673838 17031 leader_election.cc:304] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.687099 17183 raft_consensus.cc:740] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
W20260709 13:05:40.687716 17030 leader_election.cc:343] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.687934 17183 consensus_queue.cc:260] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.690163 17183 ts_tablet_manager.cc:1434] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.001s
I20260709 13:05:40.691005 17279 raft_consensus.cc:2804] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:40.691745 17279 raft_consensus.cc:697] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:40.692169 17184 tablet_bootstrap.cc:654] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.692829 17279 consensus_queue.cc:237] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.696728 17186 ts_tablet_manager.cc:1434] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 13:05:40.698076 17183 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:40.697999 17186 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:40.704699 17183 tablet_bootstrap.cc:654] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.707504 17185 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:40.708114 17185 ts_tablet_manager.cc:1403] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.159s	user 0.021s	sys 0.006s
I20260709 13:05:40.711089 17186 tablet_bootstrap.cc:654] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.710927 17185 raft_consensus.cc:359] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.711673 17185 raft_consensus.cc:385] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.711980 17185 raft_consensus.cc:740] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.712656 17185 consensus_queue.cc:260] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.714775 17185 ts_tablet_manager.cc:1434] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:05:40.717245 17185 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:40.724066 17185 tablet_bootstrap.cc:654] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.728910 17183 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:40.729430 17183 ts_tablet_manager.cc:1403] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.032s	user 0.007s	sys 0.006s
I20260709 13:05:40.732141 17183 raft_consensus.cc:359] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.732789 17183 raft_consensus.cc:385] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.733034 17183 raft_consensus.cc:740] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.733798 17183 consensus_queue.cc:260] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.736332 17183 ts_tablet_manager.cc:1434] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:05:40.737332 17183 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:40.742975 16813 catalog_manager.cc:5697] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:40.747951 17186 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.748029 17185 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:40.748498 17186 ts_tablet_manager.cc:1403] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.051s	user 0.012s	sys 0.000s
I20260709 13:05:40.748610 17185 ts_tablet_manager.cc:1403] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.013s
I20260709 13:05:40.752085 17184 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:40.752578 17184 ts_tablet_manager.cc:1403] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.075s	user 0.005s	sys 0.010s
I20260709 13:05:40.754056 17183 tablet_bootstrap.cc:654] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.755203 17184 raft_consensus.cc:359] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.755877 17184 raft_consensus.cc:385] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.756148 17184 raft_consensus.cc:740] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.751686 17186 raft_consensus.cc:359] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.756927 17186 raft_consensus.cc:385] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.757324 17186 raft_consensus.cc:740] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.756997 17184 consensus_queue.cc:260] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.758090 17186 consensus_queue.cc:260] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.761109 17186 ts_tablet_manager.cc:1434] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 13:05:40.762686 17186 tablet_bootstrap.cc:492] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:40.765167 17370 raft_consensus.cc:493] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.766386 17370 raft_consensus.cc:515] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.769186 17370 leader_election.cc:290] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:40.770540 17375 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.771060 17375 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.773059 17375 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:40.774559 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.775305 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:40.775568 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:40.775990 17064 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:40.776194 17137 raft_consensus.cc:2468] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:40.776273 17186 tablet_bootstrap.cc:654] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.776980 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.777865 16954 leader_election.cc:304] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:40.778355 17102 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.775321 16914 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
W20260709 13:05:40.779888 16957 leader_election.cc:343] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.780848 17375 raft_consensus.cc:2804] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.781890 17375 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:40.782308 17375 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.781503 17370 raft_consensus.cc:2804] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.785835 17370 raft_consensus.cc:493] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:40.786284 17370 raft_consensus.cc:3060] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.790907 17184 ts_tablet_manager.cc:1434] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 13:05:40.793386 17370 raft_consensus.cc:515] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.797336 17184 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:40.801429 17271 raft_consensus.cc:493] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.802026 17271 raft_consensus.cc:515] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.798480 17185 raft_consensus.cc:359] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.804311 17271 leader_election.cc:290] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:40.801422 17279 raft_consensus.cc:493] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.805425 17279 raft_consensus.cc:515] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.807684 17184 tablet_bootstrap.cc:654] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.807628 17279 leader_election.cc:290] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:40.808548 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.809346 16914 raft_consensus.cc:2468] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:40.810138 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.811122 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.811846 16990 raft_consensus.cc:2468] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
W20260709 13:05:40.812466 17104 leader_election.cc:343] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.813763 17031 leader_election.cc:304] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.814814 17279 raft_consensus.cc:2804] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.815227 17279 raft_consensus.cc:493] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:40.815548 17279 raft_consensus.cc:3060] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.804342 17185 raft_consensus.cc:385] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.824210 17185 raft_consensus.cc:740] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.824462 17183 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:40.824958 17183 ts_tablet_manager.cc:1403] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.088s	user 0.013s	sys 0.005s
I20260709 13:05:40.824874 17185 consensus_queue.cc:260] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.827988 17183 raft_consensus.cc:359] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.828698 17183 raft_consensus.cc:385] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.828959 17183 raft_consensus.cc:740] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.829675 17183 consensus_queue.cc:260] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.838160 17185 ts_tablet_manager.cc:1434] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.089s	user 0.006s	sys 0.001s
I20260709 13:05:40.823755 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:40.840595 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:40.841267 17137 raft_consensus.cc:3060] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.843613 17184 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:40.844157 17184 ts_tablet_manager.cc:1403] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.047s	user 0.016s	sys 0.001s
I20260709 13:05:40.844837 17279 raft_consensus.cc:515] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.847285 17183 ts_tablet_manager.cc:1434] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 13:05:40.847952 17184 raft_consensus.cc:359] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.848658 17184 raft_consensus.cc:385] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.848912 17184 raft_consensus.cc:740] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
W20260709 13:05:40.846848 17105 leader_election.cc:343] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.849634 17105 leader_election.cc:304] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:40.850247 17186 tablet_bootstrap.cc:492] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.850767 17186 ts_tablet_manager.cc:1403] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.088s	user 0.003s	sys 0.009s
I20260709 13:05:40.851493 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:40.852083 16914 raft_consensus.cc:3060] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.853610 17279 leader_election.cc:290] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:40.854785 17186 raft_consensus.cc:359] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.855508 17186 raft_consensus.cc:385] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.855769 17186 raft_consensus.cc:740] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.856593 17186 consensus_queue.cc:260] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.857873 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:40.858497 16989 raft_consensus.cc:3060] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:40.858928 17183 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:40.860601 17184 consensus_queue.cc:260] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.861860 17271 raft_consensus.cc:2749] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:40.867938 17186 ts_tablet_manager.cc:1434] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 13:05:40.874660 17184 ts_tablet_manager.cc:1434] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.002s
I20260709 13:05:40.878039 17186 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:40.879360 17184 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:40.880031 17137 raft_consensus.cc:2468] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:40.881522 16954 leader_election.cc:304] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:40.882440 17324 raft_consensus.cc:2804] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:40.882999 17324 raft_consensus.cc:697] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:40.883850 17324 consensus_queue.cc:237] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:40.888564 17183 tablet_bootstrap.cc:654] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.891685 17184 tablet_bootstrap.cc:654] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.892324 17186 tablet_bootstrap.cc:654] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.904348 17184 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:40.904917 17184 ts_tablet_manager.cc:1403] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.026s	user 0.010s	sys 0.000s
I20260709 13:05:40.908842 17186 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.909832 17183 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:40.910393 17183 ts_tablet_manager.cc:1403] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.052s	user 0.009s	sys 0.005s
I20260709 13:05:40.914829 17186 ts_tablet_manager.cc:1403] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.037s	user 0.012s	sys 0.007s
I20260709 13:05:40.918133 16914 raft_consensus.cc:2468] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:40.918582 17271 consensus_queue.cc:1048] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:40.919665 17031 leader_election.cc:304] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.920120 16814 catalog_manager.cc:5697] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:40.922230 16989 raft_consensus.cc:2468] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:40.923138 17379 raft_consensus.cc:2804] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:40.927654 17183 raft_consensus.cc:359] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.936964 17183 raft_consensus.cc:385] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.937321 17183 raft_consensus.cc:740] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.938410 17183 consensus_queue.cc:260] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.933970 17184 raft_consensus.cc:359] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.934605 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:40.939476 17184 raft_consensus.cc:385] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.939850 17184 raft_consensus.cc:740] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.935549 17186 raft_consensus.cc:359] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.940593 17184 consensus_queue.cc:260] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.941067 17186 raft_consensus.cc:385] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.941677 17186 raft_consensus.cc:740] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.923247 17370 leader_election.cc:290] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:40.952322 16957 leader_election.cc:343] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:40.936280 17185 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:40.942958 17186 consensus_queue.cc:260] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:40.957814 17246 consensus_queue.cc:1048] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:40.958279 17186 ts_tablet_manager.cc:1434] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 13:05:40.959376 17186 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:40.966657 17186 tablet_bootstrap.cc:654] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.975039 17183 ts_tablet_manager.cc:1434] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.064s	user 0.007s	sys 0.000s
I20260709 13:05:40.977742 17184 ts_tablet_manager.cc:1434] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.072s	user 0.009s	sys 0.001s
I20260709 13:05:40.978876 17184 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:40.980086 17186 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:40.980615 17186 ts_tablet_manager.cc:1403] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.000s
I20260709 13:05:40.983639 17186 raft_consensus.cc:359] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.984360 17186 raft_consensus.cc:385] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:40.984651 17186 raft_consensus.cc:740] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:40.984830 17184 tablet_bootstrap.cc:654] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:40.985376 17186 consensus_queue.cc:260] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:40.988293 17186 ts_tablet_manager.cc:1434] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 13:05:40.988683 17370 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.989368 17246 raft_consensus.cc:493] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.990036 17186 tablet_bootstrap.cc:492] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:40.990270 17246 raft_consensus.cc:515] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.992470 17246 leader_election.cc:290] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:40.992775 17255 raft_consensus.cc:493] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:40.993330 17255 raft_consensus.cc:515] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:40.995515 17255 leader_election.cc:290] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:40.995690 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:40.996409 16990 raft_consensus.cc:2468] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:40.997588 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:40.997853 17104 leader_election.cc:304] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:40.998313 17064 raft_consensus.cc:2468] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:40.999027 17246 raft_consensus.cc:2804] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:40.999433 17246 raft_consensus.cc:493] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:40.999610 16954 leader_election.cc:304] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:40.999797 17246 raft_consensus.cc:3060] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.000542 17255 raft_consensus.cc:2804] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:41.000777 17186 tablet_bootstrap.cc:654] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.000942 17255 raft_consensus.cc:493] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.001502 17255 raft_consensus.cc:3060] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.003201 17376 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.003765 17376 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.006397 17376 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:41.008047 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.008221 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.009110 17064 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:41.010318 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:41.011883 17255 raft_consensus.cc:515] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.012400 17102 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:41.013322 17186 tablet_bootstrap.cc:492] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:41.013450 17376 raft_consensus.cc:2804] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:41.013773 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.013965 17186 ts_tablet_manager.cc:1403] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.008s
I20260709 13:05:41.014498 17062 raft_consensus.cc:2468] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
W20260709 13:05:41.014796 17105 leader_election.cc:343] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.017005 17186 raft_consensus.cc:359] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.017813 17186 raft_consensus.cc:385] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.014043 17376 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.018138 17186 raft_consensus.cc:740] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.018379 17376 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.019146 17186 consensus_queue.cc:260] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.021831 17186 ts_tablet_manager.cc:1434] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 13:05:41.022486 17183 tablet_bootstrap.cc:492] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:41.022804 17186 tablet_bootstrap.cc:492] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:41.030423 17255 leader_election.cc:290] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.033670 17186 tablet_bootstrap.cc:654] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.035049 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:41.035723 17062 raft_consensus.cc:3060] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.037755 17370 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.048380 17370 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:40.795898 17375 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.049536 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.062582 17064 raft_consensus.cc:2468] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:41.064030 16954 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:41.064934 17407 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.051605 17379 raft_consensus.cc:697] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:41.069038 17185 tablet_bootstrap.cc:654] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.069593 17379 consensus_queue.cc:237] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.052465 17184 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:41.050989 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:41.058276 17376 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.065389 17375 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:41.065622 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:41.072396 17184 ts_tablet_manager.cc:1403] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.094s	user 0.016s	sys 0.011s
I20260709 13:05:41.072427 17064 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.075438 17184 raft_consensus.cc:359] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.076161 17184 raft_consensus.cc:385] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.076184 17376 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:41.076866 16957 leader_election.cc:343] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.066093 17407 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.076439 17184 raft_consensus.cc:740] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.078650 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:41.078522 17184 consensus_queue.cc:260] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.066627 17255 raft_consensus.cc:2804] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:41.068339 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:41.061197 17246 raft_consensus.cc:515] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.079763 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:41.080617 16989 raft_consensus.cc:2393] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:41.082079 17139 raft_consensus.cc:2393] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:41.082402 17407 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.086771 17246 leader_election.cc:290] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:41.089798 17353 raft_consensus.cc:493] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.087916 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.090343 17353 raft_consensus.cc:515] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.091434 17139 raft_consensus.cc:2393] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:41.092273 17064 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:41.093274 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.079775 17255 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.093807 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:41.094059 17138 raft_consensus.cc:2468] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:41.094224 17255 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.094619 16989 raft_consensus.cc:2393] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:41.095367 17028 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:41.096204 17407 raft_consensus.cc:3060] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.097164 16879 leader_election.cc:304] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:41.098145 17352 raft_consensus.cc:2804] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:41.098591 17352 raft_consensus.cc:493] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.098981 17352 raft_consensus.cc:3060] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.102823 16914 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:41.109959 17105 leader_election.cc:343] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.119084 17353 leader_election.cc:290] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:41.119299 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:41.120011 16989 raft_consensus.cc:2468] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:41.122396 17102 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:41.124150 17375 raft_consensus.cc:2804] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:41.124672 17375 raft_consensus.cc:697] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:41.125551 17375 consensus_queue.cc:237] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.125775 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:41.127072 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:41.127044 17137 raft_consensus.cc:2393] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:41.130530 17062 raft_consensus.cc:2468] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:41.134507 17102 leader_election.cc:304] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:41.137430 17376 raft_consensus.cc:2749] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.125766 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:41.126012 17183 tablet_bootstrap.cc:654] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.138339 17061 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.140621 16954 leader_election.cc:304] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:41.142151 17186 tablet_bootstrap.cc:492] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:41.142356 17407 raft_consensus.cc:2749] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.142848 17186 ts_tablet_manager.cc:1403] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.120s	user 0.011s	sys 0.000s
I20260709 13:05:41.143253 17324 raft_consensus.cc:2804] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:41.143929 17324 raft_consensus.cc:697] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:41.144966 17324 consensus_queue.cc:237] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.082125 17184 ts_tablet_manager.cc:1434] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:05:41.150486 17184 tablet_bootstrap.cc:492] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:41.154642 17186 raft_consensus.cc:359] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.155416 17186 raft_consensus.cc:385] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.155704 17186 raft_consensus.cc:740] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.156338 17186 consensus_queue.cc:260] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.159610 17061 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:41.162658 17183 tablet_bootstrap.cc:492] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:41.163292 17183 ts_tablet_manager.cc:1403] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.141s	user 0.005s	sys 0.007s
I20260709 13:05:41.168076 16914 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:41.168395 17102 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:41.166829 17183 raft_consensus.cc:359] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.169999 17375 raft_consensus.cc:2804] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:41.170282 17183 raft_consensus.cc:385] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.170789 17375 raft_consensus.cc:697] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:41.170672 17183 raft_consensus.cc:740] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.172194 17183 consensus_queue.cc:260] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.173532 17352 raft_consensus.cc:515] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.172037 17375 consensus_queue.cc:237] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.175396 17279 consensus_queue.cc:1048] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:41.176231 17186 ts_tablet_manager.cc:1434] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.000s
I20260709 13:05:41.177343 17186 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:41.159641 17255 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.185515 16814 catalog_manager.cc:5697] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:41.186497 17255 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:41.187043 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:41.187844 17185 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:41.188344 17185 ts_tablet_manager.cc:1403] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.252s	user 0.032s	sys 0.023s
I20260709 13:05:41.189538 17186 tablet_bootstrap.cc:654] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.188948 16813 catalog_manager.cc:5697] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
W20260709 13:05:41.189431 16957 leader_election.cc:343] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.191247 17185 raft_consensus.cc:359] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.191977 17185 raft_consensus.cc:385] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.192356 17185 raft_consensus.cc:740] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.200796 17184 tablet_bootstrap.cc:654] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.200950 17186 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:41.201560 17186 ts_tablet_manager.cc:1403] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.006s
I20260709 13:05:41.193228 17185 consensus_queue.cc:260] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.204684 17186 raft_consensus.cc:359] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.205564 17186 raft_consensus.cc:385] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.205906 17186 raft_consensus.cc:740] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.206818 17186 consensus_queue.cc:260] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.209512 17186 ts_tablet_manager.cc:1434] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:05:41.210610 17186 tablet_bootstrap.cc:492] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:41.213121 17183 ts_tablet_manager.cc:1434] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.001s
I20260709 13:05:41.214316 17183 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:41.220028 17186 tablet_bootstrap.cc:654] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.220752 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:41.221423 17064 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.216785 17185 ts_tablet_manager.cc:1434] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.004s
I20260709 13:05:41.220783 17183 tablet_bootstrap.cc:654] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.234802 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:41.253135 16989 raft_consensus.cc:3060] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.237164 17185 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:41.258585 17375 raft_consensus.cc:493] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.259159 17375 raft_consensus.cc:515] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.240010 17379 raft_consensus.cc:493] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.263682 17185 tablet_bootstrap.cc:654] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.263509 17379 raft_consensus.cc:515] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.264168 17375 leader_election.cc:290] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:41.244437 16990 raft_consensus.cc:3060] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.265836 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:41.266335 17379 leader_election.cc:290] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:41.243953 17184 tablet_bootstrap.cc:492] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:41.267724 17184 ts_tablet_manager.cc:1403] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.117s	user 0.008s	sys 0.011s
I20260709 13:05:41.267575 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.268234 17138 raft_consensus.cc:2468] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:41.269313 17028 leader_election.cc:304] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:41.233016 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:41.270277 17105 leader_election.cc:343] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.270771 17137 raft_consensus.cc:3060] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.247493 17186 tablet_bootstrap.cc:492] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:41.271613 17186 ts_tablet_manager.cc:1403] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.061s	user 0.022s	sys 0.011s
I20260709 13:05:41.239988 17352 leader_election.cc:290] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:41.274005 17279 raft_consensus.cc:2804] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:41.275964 17279 raft_consensus.cc:493] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.276427 17279 raft_consensus.cc:3060] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.274887 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:41.230207 17064 raft_consensus.cc:2468] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:41.275012 17186 raft_consensus.cc:359] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.279541 17186 raft_consensus.cc:385] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.279942 17186 raft_consensus.cc:740] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
W20260709 13:05:41.280179 17031 leader_election.cc:343] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.280812 17186 consensus_queue.cc:260] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.283253 17184 raft_consensus.cc:359] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.283497 16811 catalog_manager.cc:5697] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:41.286391 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:41.288187 17379 consensus_queue.cc:1048] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.235069 17407 raft_consensus.cc:493] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.292429 17426 raft_consensus.cc:493] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.298758 17407 raft_consensus.cc:515] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.300256 17426 raft_consensus.cc:515] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.302644 17407 leader_election.cc:290] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.303040 17426 leader_election.cc:290] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.292683 17279 raft_consensus.cc:515] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.294806 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.295362 17184 raft_consensus.cc:385] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.344341 17184 raft_consensus.cc:740] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.345149 17184 consensus_queue.cc:260] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.317348 17183 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:41.346433 17183 ts_tablet_manager.cc:1403] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.132s	user 0.016s	sys 0.031s
I20260709 13:05:41.347198 17186 ts_tablet_manager.cc:1434] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.075s	user 0.017s	sys 0.027s
I20260709 13:05:41.348330 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:41.348958 17136 raft_consensus.cc:3060] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.349431 17184 ts_tablet_manager.cc:1434] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.081s	user 0.006s	sys 0.001s
I20260709 13:05:41.302893 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:41.356827 16991 raft_consensus.cc:2468] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:41.357214 17184 tablet_bootstrap.cc:492] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
W20260709 13:05:41.361686 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 75860.1us
I20260709 13:05:41.362322 17186 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:41.363523 17030 leader_election.cc:304] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:41.363902 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:41.316326 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:41.304585 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.367194 17379 raft_consensus.cc:2804] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:41.367717 17379 raft_consensus.cc:493] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.368075 17379 raft_consensus.cc:3060] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:41.368351 17030 leader_election.cc:343] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.305567 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.369053 17186 tablet_bootstrap.cc:654] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:41.370222 17028 leader_election.cc:343] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.370685 17028 leader_election.cc:304] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:41.373102 17426 raft_consensus.cc:2749] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.333029 16813 catalog_manager.cc:5697] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:41.376010 17136 raft_consensus.cc:2468] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
W20260709 13:05:41.378430 17031 leader_election.cc:343] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.337007 17137 raft_consensus.cc:2468] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:41.345719 17279 leader_election.cc:290] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:41.385739 16879 leader_election.cc:304] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:41.386762 17352 raft_consensus.cc:2804] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:41.387336 17352 raft_consensus.cc:697] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:41.349537 17183 raft_consensus.cc:359] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.388242 17183 raft_consensus.cc:385] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.388612 17183 raft_consensus.cc:740] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.351953 17064 raft_consensus.cc:2393] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:41.388268 17352 consensus_queue.cc:237] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.354182 17185 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:41.390383 17185 ts_tablet_manager.cc:1403] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.154s	user 0.006s	sys 0.009s
I20260709 13:05:41.390827 17183 consensus_queue.cc:260] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.393420 17183 ts_tablet_manager.cc:1434] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.000s
I20260709 13:05:41.393232 17185 raft_consensus.cc:359] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.393728 17102 leader_election.cc:304] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:41.394074 17185 raft_consensus.cc:385] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.394408 17185 raft_consensus.cc:740] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.394588 17183 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:41.395283 17375 raft_consensus.cc:2749] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.395118 17185 consensus_queue.cc:260] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.400643 17418 raft_consensus.cc:493] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.401292 17418 raft_consensus.cc:515] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:41.402289 17183 tablet_bootstrap.cc:654] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.331086 16989 raft_consensus.cc:2468] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:41.293785 16954 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:41.406363 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 112666us
I20260709 13:05:41.397222 17185 ts_tablet_manager.cc:1434] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.001s
I20260709 13:05:41.408141 17185 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:41.411857 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:41.413702 17418 leader_election.cc:290] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:41.415587 17255 raft_consensus.cc:2804] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:41.416357 17255 raft_consensus.cc:697] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:41.417518 17255 consensus_queue.cc:237] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.419670 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.420778 17137 raft_consensus.cc:2393] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:41.421066 16912 raft_consensus.cc:3060] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.421990 17028 leader_election.cc:304] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:41.425594 17426 raft_consensus.cc:2804] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:41.425891 16879 leader_election.cc:304] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:41.427076 17426 raft_consensus.cc:697] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:41.428040 17426 consensus_queue.cc:237] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.440232 17418 raft_consensus.cc:2749] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.373569 17138 raft_consensus.cc:2468] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:41.380409 17379 raft_consensus.cc:515] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.443009 17330 consensus_queue.cc:1048] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.458290 17379 leader_election.cc:290] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.415077 17185 tablet_bootstrap.cc:654] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.462736 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:41.466419 17324 consensus_queue.cc:1048] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:41.471968 16989 raft_consensus.cc:3060] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.468881 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:41.479722 17139 raft_consensus.cc:3060] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.487551 17184 tablet_bootstrap.cc:654] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.488653 16811 catalog_manager.cc:5697] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:41.491812 17186 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:41.492363 17186 ts_tablet_manager.cc:1403] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.130s	user 0.009s	sys 0.008s
I20260709 13:05:41.495226 17186 raft_consensus.cc:359] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.495935 17186 raft_consensus.cc:385] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.496204 17186 raft_consensus.cc:740] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.497120 17186 consensus_queue.cc:260] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.500061 17186 ts_tablet_manager.cc:1434] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:05:41.472023 16812 catalog_manager.cc:5697] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:41.501077 17186 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:41.507874 17186 tablet_bootstrap.cc:654] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.515645 16813 catalog_manager.cc:5697] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:41.517355 17184 tablet_bootstrap.cc:492] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:41.517992 17184 ts_tablet_manager.cc:1403] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.161s	user 0.012s	sys 0.000s
I20260709 13:05:41.518049 17279 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.518754 17279 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.517429 17379 consensus_queue.cc:1048] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:05:41.520748 17184 raft_consensus.cc:359] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.521456 17184 raft_consensus.cc:385] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.521713 17184 raft_consensus.cc:740] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.522379 17184 consensus_queue.cc:260] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.522583 17279 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:41.523903 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:41.529870 17184 ts_tablet_manager.cc:1434] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
W20260709 13:05:41.531898 17030 leader_election.cc:343] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.541139 17184 tablet_bootstrap.cc:492] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:41.548009 17184 tablet_bootstrap.cc:654] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.557512 17279 consensus_queue.cc:1048] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.558525 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:41.559857 17324 consensus_queue.cc:1048] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:05:41.561187 17031 leader_election.cc:343] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.561725 17031 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:41.563484 17279 raft_consensus.cc:2749] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.583307 17324 consensus_queue.cc:1048] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.591324 17184 tablet_bootstrap.cc:492] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:41.591856 17184 ts_tablet_manager.cc:1403] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.051s	user 0.010s	sys 0.004s
I20260709 13:05:41.594724 17184 raft_consensus.cc:359] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.595440 17184 raft_consensus.cc:385] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.595713 17184 raft_consensus.cc:740] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.596295 17184 consensus_queue.cc:260] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.614216 17184 ts_tablet_manager.cc:1434] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.001s
I20260709 13:05:41.615276 16914 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.616899 17184 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:41.630366 17324 raft_consensus.cc:493] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.631363 17324 raft_consensus.cc:515] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.639026 17324 leader_election.cc:290] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.639703 17184 tablet_bootstrap.cc:654] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.647704 17379 raft_consensus.cc:493] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.648423 17379 raft_consensus.cc:515] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.651388 17379 leader_election.cc:290] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.653424 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.654310 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:41.655297 17028 leader_election.cc:343] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:41.656206 17030 leader_election.cc:343] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.656816 17030 leader_election.cc:304] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:41.658201 17379 raft_consensus.cc:2749] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.659444 17183 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:41.660040 17183 ts_tablet_manager.cc:1403] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.266s	user 0.102s	sys 0.099s
I20260709 13:05:41.660706 17418 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.661378 17184 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:41.661267 17418 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.661911 17184 ts_tablet_manager.cc:1403] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.000s
I20260709 13:05:41.663997 17183 raft_consensus.cc:359] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.664819 17183 raft_consensus.cc:385] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.665191 17183 raft_consensus.cc:740] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.666096 17183 consensus_queue.cc:260] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.669453 17183 ts_tablet_manager.cc:1434] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:05:41.670576 17418 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.670382 17376 consensus_queue.cc:1048] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.670439 17183 tablet_bootstrap.cc:492] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:41.664499 17184 raft_consensus.cc:359] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.672673 17184 raft_consensus.cc:385] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.672962 17184 raft_consensus.cc:740] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.674407 17184 consensus_queue.cc:260] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.676522 17184 ts_tablet_manager.cc:1434] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260709 13:05:41.677163 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.678153 17183 tablet_bootstrap.cc:654] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.680781 17184 tablet_bootstrap.cc:492] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:41.682751 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.685546 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.686542 17062 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:41.688171 16879 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:41.689924 17418 raft_consensus.cc:2804] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:41.690502 17418 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.691031 17418 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:41.694957 17186 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:41.695580 17186 ts_tablet_manager.cc:1403] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.195s	user 0.023s	sys 0.029s
I20260709 13:05:41.698726 17186 raft_consensus.cc:359] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.699496 17186 raft_consensus.cc:385] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.700038 17186 raft_consensus.cc:740] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.659444 17185 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:41.704344 17185 ts_tablet_manager.cc:1403] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.296s	user 0.008s	sys 0.004s
I20260709 13:05:41.701012 17186 consensus_queue.cc:260] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.709813 17184 tablet_bootstrap.cc:654] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.699265 16989 raft_consensus.cc:2468] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:41.729605 17376 consensus_queue.cc:1048] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.739727 17186 ts_tablet_manager.cc:1434] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.002s
I20260709 13:05:41.740819 17186 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:41.742918 17418 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.734221 17139 raft_consensus.cc:2468] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:41.745239 17418 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:41.750512 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:41.754115 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.774995 17186 tablet_bootstrap.cc:654] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.778373 17185 raft_consensus.cc:359] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.779184 17185 raft_consensus.cc:385] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.779459 17185 raft_consensus.cc:740] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.780689 17185 consensus_queue.cc:260] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.779946 17030 leader_election.cc:304] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:41.773856 17064 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:41.787429 16913 raft_consensus.cc:1240] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:05:41.789011 17426 raft_consensus.cc:2804] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:41.789856 17426 raft_consensus.cc:697] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:41.790792 17426 consensus_queue.cc:237] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.794415 17375 consensus_queue.cc:1048] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.774578 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:41.795585 17330 raft_consensus.cc:493] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.801527 17330 raft_consensus.cc:515] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.802552 17451 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.803658 17330 leader_election.cc:290] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:41.804776 16954 leader_election.cc:343] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:41.805899 16954 leader_election.cc:343] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.806078 17064 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:41.806344 16954 leader_election.cc:304] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:41.803269 17451 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.807701 17185 ts_tablet_manager.cc:1434] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.103s	user 0.006s	sys 0.001s
I20260709 13:05:41.812712 17255 raft_consensus.cc:2749] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.814201 17185 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:41.813732 17062 raft_consensus.cc:1240] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:41.825973 16879 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:41.827685 17352 raft_consensus.cc:2804] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:41.828372 17352 raft_consensus.cc:697] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:41.832067 17451 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:41.838361 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.843461 17352 consensus_queue.cc:237] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.859925 17183 tablet_bootstrap.cc:492] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:41.860476 17183 ts_tablet_manager.cc:1403] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.190s	user 0.031s	sys 0.043s
I20260709 13:05:41.844964 17376 consensus_queue.cc:1059] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20260709 13:05:41.859535 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 70012.5us
W20260709 13:05:41.858520 17104 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 68388.8us
I20260709 13:05:41.877928 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.873906 17451 raft_consensus.cc:3055] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:41.888401 17451 raft_consensus.cc:740] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:41.889176 17451 consensus_queue.cc:260] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.890476 17451 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:41.892872 17418 consensus_queue.cc:1048] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.894263 17379 raft_consensus.cc:493] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:41.894830 17379 raft_consensus.cc:515] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:41.898891 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:41.900799 16811 catalog_manager.cc:5697] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:41.915135 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:41.916450 17061 raft_consensus.cc:2468] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:41.926364 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:41.929092 17184 tablet_bootstrap.cc:492] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:41.929759 17184 ts_tablet_manager.cc:1403] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.253s	user 0.009s	sys 0.026s
W20260709 13:05:41.930545 17031 leader_election.cc:343] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:41.884349 16954 leader_election.cc:343] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.933375 17184 raft_consensus.cc:359] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.897289 17379 leader_election.cc:290] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:41.937554 17184 raft_consensus.cc:385] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.938632 17184 raft_consensus.cc:740] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.940429 17184 consensus_queue.cc:260] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:41.940629 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 57774.3us
W20260709 13:05:41.935072 17105 leader_election.cc:343] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:41.950129 17102 leader_election.cc:343] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:41.951256 16954 leader_election.cc:304] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:41.954849 17186 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:41.955372 17186 ts_tablet_manager.cc:1403] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.215s	user 0.017s	sys 0.005s
I20260709 13:05:41.955372 17185 tablet_bootstrap.cc:654] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:41.952165 17255 consensus_queue.cc:1048] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.955701 17459 raft_consensus.cc:2804] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:41.957895 17459 raft_consensus.cc:493] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:41.958266 17459 raft_consensus.cc:3060] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:41.959828 17184 ts_tablet_manager.cc:1434] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.002s
I20260709 13:05:41.961223 17184 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:41.963460 17102 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:41.966360 17452 raft_consensus.cc:2749] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:41.969264 17183 raft_consensus.cc:359] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.970000 17183 raft_consensus.cc:385] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.972797 17461 consensus_queue.cc:1048] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:41.970253 17183 raft_consensus.cc:740] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.982913 17183 consensus_queue.cc:260] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:41.983065 17186 raft_consensus.cc:359] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.983814 17186 raft_consensus.cc:385] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:41.984082 17186 raft_consensus.cc:740] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:41.984758 17186 consensus_queue.cc:260] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:41.990408 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:41.991079 17139 raft_consensus.cc:2468] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:41.991667 17186 ts_tablet_manager.cc:1434] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.000s
I20260709 13:05:41.992720 17186 tablet_bootstrap.cc:492] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:42.003449 17330 raft_consensus.cc:493] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.006757 17330 raft_consensus.cc:515] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.008872 17330 leader_election.cc:290] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:42.016836 17183 ts_tablet_manager.cc:1434] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.156s	user 0.007s	sys 0.001s
I20260709 13:05:42.018078 17183 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:42.019253 17426 consensus_queue.cc:1048] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:42.019454 17028 leader_election.cc:304] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.020363 17379 raft_consensus.cc:2804] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.020758 17379 raft_consensus.cc:493] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.021122 17379 raft_consensus.cc:3060] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.024235 17183 tablet_bootstrap.cc:654] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.028867 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e77860bb944a5e97539267e7d0f9ba" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:42.029702 16912 raft_consensus.cc:2468] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:42.031121 16957 leader_election.cc:304] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:42.031944 16812 catalog_manager.cc:5697] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:42.035395 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e77860bb944a5e97539267e7d0f9ba" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:42.036027 16957 consensus_peers.cc:597] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:42.036139 17061 raft_consensus.cc:2468] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:42.037047 17255 raft_consensus.cc:2804] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.037469 17255 raft_consensus.cc:493] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.037817 17255 raft_consensus.cc:3060] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.041576 17186 tablet_bootstrap.cc:654] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.036309 17379 raft_consensus.cc:515] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:42.061781 17031 consensus_peers.cc:597] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:42.072958 17255 raft_consensus.cc:515] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.076881 17379 leader_election.cc:290] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:42.078027 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
W20260709 13:05:42.080521 17031 leader_election.cc:343] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.102820 17184 tablet_bootstrap.cc:654] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.109193 17456 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.110096 17456 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.114537 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e77860bb944a5e97539267e7d0f9ba" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:42.115263 16916 raft_consensus.cc:3060] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.123988 17255 leader_election.cc:290] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:42.129910 17255 raft_consensus.cc:493] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.130990 17185 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:42.130403 17255 raft_consensus.cc:515] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.131543 17185 ts_tablet_manager.cc:1403] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.321s	user 0.025s	sys 0.019s
I20260709 13:05:42.133024 17255 leader_election.cc:290] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:42.135447 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:42.137254 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e77860bb944a5e97539267e7d0f9ba" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:42.137843 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:42.141008 17186 tablet_bootstrap.cc:492] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:42.152458 17186 ts_tablet_manager.cc:1403] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.160s	user 0.016s	sys 0.026s
I20260709 13:05:42.141624 17185 raft_consensus.cc:359] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.143862 17183 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:42.113176 17456 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:42.151098 17064 raft_consensus.cc:3060] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.154556 17185 raft_consensus.cc:385] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.155023 17183 ts_tablet_manager.cc:1403] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.137s	user 0.020s	sys 0.014s
I20260709 13:05:42.155961 17063 raft_consensus.cc:2468] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:42.155138 17185 raft_consensus.cc:740] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.159689 17185 consensus_queue.cc:260] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.161602 17064 raft_consensus.cc:2468] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:42.163319 17186 raft_consensus.cc:359] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.163830 17186 raft_consensus.cc:385] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.164117 17186 raft_consensus.cc:740] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.166489 17185 ts_tablet_manager.cc:1434] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.004s
I20260709 13:05:42.167274 17185 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:42.172720 17183 raft_consensus.cc:359] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.177894 17183 raft_consensus.cc:385] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.178243 17183 raft_consensus.cc:740] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.178900 17183 consensus_queue.cc:260] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.174512 16916 raft_consensus.cc:2468] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:42.161604 17459 raft_consensus.cc:515] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:42.165174 16954 leader_election.cc:343] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.181226 16954 leader_election.cc:304] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:42.182008 17459 leader_election.cc:290] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:42.182667 17459 raft_consensus.cc:2804] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.182724 16954 leader_election.cc:304] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:42.185717 17186 consensus_queue.cc:260] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.186077 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:42.186787 16989 raft_consensus.cc:2468] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:42.187422 17186 ts_tablet_manager.cc:1434] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.000s
I20260709 13:05:42.188434 17186 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:42.189278 17470 raft_consensus.cc:2804] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.189975 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:42.190644 17062 raft_consensus.cc:3060] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.192866 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:42.193776 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:42.194453 17063 raft_consensus.cc:2468] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
W20260709 13:05:42.194978 16954 leader_election.cc:343] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.199453 17185 tablet_bootstrap.cc:654] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.190119 17470 raft_consensus.cc:697] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:42.202986 17470 consensus_queue.cc:237] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.183131 17459 raft_consensus.cc:493] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.206890 17459 raft_consensus.cc:3060] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.207337 17062 raft_consensus.cc:2468] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:42.209897 16954 leader_election.cc:304] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:42.210436 17379 consensus_queue.cc:1048] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:42.211884 17186 tablet_bootstrap.cc:654] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.222229 17471 raft_consensus.cc:2804] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.223096 17471 raft_consensus.cc:697] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:42.213464 17102 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:42.216120 17183 ts_tablet_manager.cc:1434] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.061s	user 0.005s	sys 0.001s
I20260709 13:05:42.211795 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:42.225874 17137 raft_consensus.cc:3060] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.226337 17451 raft_consensus.cc:2804] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:42.226796 17451 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.227180 17451 raft_consensus.cc:3060] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:42.228615 17184 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:42.229147 17184 ts_tablet_manager.cc:1403] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.268s	user 0.017s	sys 0.003s
I20260709 13:05:42.231928 17184 raft_consensus.cc:359] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.232750 17184 raft_consensus.cc:385] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.233038 17184 raft_consensus.cc:740] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.225104 17471 consensus_queue.cc:237] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.226397 17183 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:42.243976 17185 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:42.247382 17185 ts_tablet_manager.cc:1403] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.080s	user 0.013s	sys 0.000s
I20260709 13:05:42.244313 17184 consensus_queue.cc:260] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.245256 17186 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:42.251998 17184 ts_tablet_manager.cc:1434] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 13:05:42.260711 17184 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:42.247537 17137 raft_consensus.cc:2468] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:42.250775 17186 ts_tablet_manager.cc:1403] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.063s	user 0.011s	sys 0.001s
I20260709 13:05:42.250284 17185 raft_consensus.cc:359] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.264767 17185 raft_consensus.cc:385] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.265096 17185 raft_consensus.cc:740] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.265264 17028 leader_election.cc:304] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.265846 17185 consensus_queue.cc:260] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.266629 17379 raft_consensus.cc:2804] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.267120 17379 raft_consensus.cc:697] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:42.267225 17186 raft_consensus.cc:359] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.268002 17186 raft_consensus.cc:385] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.268314 17186 raft_consensus.cc:740] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.268020 17379 consensus_queue.cc:237] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.269413 17186 consensus_queue.cc:260] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.268636 17185 ts_tablet_manager.cc:1434] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 13:05:42.271464 17185 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:42.274346 17186 ts_tablet_manager.cc:1434] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:05:42.275404 17186 tablet_bootstrap.cc:492] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:42.276160 17184 tablet_bootstrap.cc:654] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.279829 17459 raft_consensus.cc:515] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.286959 17459 leader_election.cc:290] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:42.296350 17185 tablet_bootstrap.cc:654] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.296525 17184 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:42.297118 17184 ts_tablet_manager.cc:1403] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.037s	user 0.012s	sys 0.000s
I20260709 13:05:42.297511 16813 catalog_manager.cc:5697] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:42.301996 17062 raft_consensus.cc:1275] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:05:42.303915 17352 consensus_queue.cc:1048] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:42.254662 17451 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.305572 17184 raft_consensus.cc:359] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.318019 17184 raft_consensus.cc:385] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.320653 17184 raft_consensus.cc:740] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.320083 17186 tablet_bootstrap.cc:654] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.318039 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:42.321529 17184 consensus_queue.cc:260] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.321981 17062 raft_consensus.cc:3060] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.323953 17184 ts_tablet_manager.cc:1434] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 13:05:42.329643 17184 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:42.330096 17279 consensus_queue.cc:1048] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:42.332459 17137 raft_consensus.cc:1275] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 13:05:42.334625 17137 pending_rounds.cc:85] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: Aborting all ops after (but not including) 0
I20260709 13:05:42.334995 17137 pending_rounds.cc:107] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 13:05:42.335338 17137 raft_consensus.cc:2889] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:05:42.318411 17352 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.340071 17421 consensus_queue.cc:1048] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:42.338402 17352 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.342635 17352 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:42.344601 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:42.345430 17137 raft_consensus.cc:2468] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:42.350919 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:42.354579 16879 leader_election.cc:343] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.357596 16879 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:42.360664 17062 raft_consensus.cc:2468] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:42.361838 17421 raft_consensus.cc:2804] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.365126 17421 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.365676 17421 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.368703 17456 raft_consensus.cc:493] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.369491 17456 raft_consensus.cc:515] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.375420 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:42.376929 17183 tablet_bootstrap.cc:654] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:42.377453 17105 leader_election.cc:343] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.358004 17184 tablet_bootstrap.cc:654] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.397572 17452 raft_consensus.cc:493] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.397761 17455 raft_consensus.cc:493] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.398204 17452 raft_consensus.cc:515] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.398350 17455 raft_consensus.cc:515] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.402741 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:42.403412 17456 leader_election.cc:290] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:42.401165 17455 leader_election.cc:290] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:42.416205 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:42.416456 17352 raft_consensus.cc:493] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.417023 17352 raft_consensus.cc:515] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:42.417654 17105 leader_election.cc:343] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.418857 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:42.419270 17352 leader_election.cc:290] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:42.419626 17062 raft_consensus.cc:2468] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:42.420884 17102 leader_election.cc:304] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:42.421072 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:42.421370 17186 tablet_bootstrap.cc:492] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:42.421777 17455 raft_consensus.cc:2804] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.421944 17186 ts_tablet_manager.cc:1403] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.147s	user 0.020s	sys 0.011s
I20260709 13:05:42.422186 17455 raft_consensus.cc:493] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.422569 17455 raft_consensus.cc:3060] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.423136 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:42.424152 17421 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.424752 16879 leader_election.cc:304] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:42.426071 17352 raft_consensus.cc:2749] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:42.424876 17186 raft_consensus.cc:359] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.427246 17186 raft_consensus.cc:385] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.427495 17186 raft_consensus.cc:740] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.428362 17186 consensus_queue.cc:260] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.436450 16811 catalog_manager.cc:5697] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:42.433646 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:42.441365 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:42.441964 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:42.442165 16991 raft_consensus.cc:2468] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:42.441565 16990 raft_consensus.cc:3060] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:42.442626 16988 raft_consensus.cc:2468] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:42.446993 17104 leader_election.cc:304] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.448022 17452 leader_election.cc:290] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:42.448673 17452 raft_consensus.cc:2804] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.449054 17452 raft_consensus.cc:493] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.449388 17452 raft_consensus.cc:3060] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.462975 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:42.463712 17064 raft_consensus.cc:3060] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:42.464411 17421 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:42.466585 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:42.453692 16990 raft_consensus.cc:2468] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:42.468506 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:42.470755 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:42.478317 17064 raft_consensus.cc:2468] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:42.480118 17186 ts_tablet_manager.cc:1434] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.058s	user 0.007s	sys 0.000s
I20260709 13:05:42.483822 17186 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:42.456331 17451 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:42.473385 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:42.485359 16954 leader_election.cc:343] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.490779 17137 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.491758 16954 leader_election.cc:304] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:42.492564 17184 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:42.492713 17459 raft_consensus.cc:2804] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.493160 17184 ts_tablet_manager.cc:1403] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.164s	user 0.011s	sys 0.001s
I20260709 13:05:42.490814 17452 raft_consensus.cc:515] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.501156 17452 leader_election.cc:290] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:42.495107 17102 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:42.503378 17454 raft_consensus.cc:2804] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:42.504009 17454 raft_consensus.cc:697] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:42.504618 17184 raft_consensus.cc:359] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:42.505172 17104 leader_election.cc:343] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.505023 17454 consensus_queue.cc:237] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.506533 17428 raft_consensus.cc:493] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.495523 17459 raft_consensus.cc:697] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:42.510341 17104 leader_election.cc:304] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:42.510627 17428 raft_consensus.cc:515] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.511137 17459 consensus_queue.cc:237] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.512714 17456 raft_consensus.cc:2749] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:42.513705 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:42.514487 16988 raft_consensus.cc:3060] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.514616 17184 raft_consensus.cc:385] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.516534 17184 raft_consensus.cc:740] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.516917 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:42.515547 17428 leader_election.cc:290] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:42.515666 16913 raft_consensus.cc:3060] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.518085 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:42.517383 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:42.519979 17185 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
W20260709 13:05:42.523962 17031 leader_election.cc:343] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:42.524857 17105 leader_election.cc:343] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.531677 17185 ts_tablet_manager.cc:1403] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.260s	user 0.020s	sys 0.028s
W20260709 13:05:42.487550 16879 leader_election.cc:343] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:42.534111 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 48583us
I20260709 13:05:42.540576 17186 tablet_bootstrap.cc:654] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.517376 17184 consensus_queue.cc:260] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.543362 16989 raft_consensus.cc:2468] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:42.548293 17030 leader_election.cc:304] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.549290 17379 raft_consensus.cc:2804] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.549947 17379 raft_consensus.cc:493] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.550670 17379 raft_consensus.cc:3060] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.549875 16813 catalog_manager.cc:5697] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:42.551949 17184 ts_tablet_manager.cc:1434] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.000s
I20260709 13:05:42.554780 17137 raft_consensus.cc:2468] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:42.556263 16879 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:42.557389 17421 raft_consensus.cc:2804] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.557947 17421 raft_consensus.cc:697] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:42.558976 17421 consensus_queue.cc:237] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.561646 17183 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:42.562167 17183 ts_tablet_manager.cc:1403] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.336s	user 0.027s	sys 0.000s
I20260709 13:05:42.563004 17184 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:42.568481 17455 raft_consensus.cc:515] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.586632 17185 raft_consensus.cc:359] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.587769 17185 raft_consensus.cc:385] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.588131 17185 raft_consensus.cc:740] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.591089 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:42.591670 17455 leader_election.cc:290] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:42.591948 16990 raft_consensus.cc:3060] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.603065 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:42.603770 17062 raft_consensus.cc:3060] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.599314 16811 catalog_manager.cc:5697] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 2, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:42.605201 17183 raft_consensus.cc:359] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.606297 17183 raft_consensus.cc:385] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.606666 17183 raft_consensus.cc:740] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.608079 17183 consensus_queue.cc:260] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.610654 16990 raft_consensus.cc:2468] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:42.611186 17183 ts_tablet_manager.cc:1434] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.000s
I20260709 13:05:42.612267 17183 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:42.617077 17104 leader_election.cc:304] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:42.618518 17455 raft_consensus.cc:2804] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.619059 17455 raft_consensus.cc:697] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:42.619266 17330 consensus_queue.cc:1048] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:42.620200 17455 consensus_queue.cc:237] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.597555 17379 raft_consensus.cc:515] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.589015 17185 consensus_queue.cc:260] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.639876 17185 ts_tablet_manager.cc:1434] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.106s	user 0.007s	sys 0.002s
I20260709 13:05:42.640875 17185 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:42.613963 16988 raft_consensus.cc:2468] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:42.655380 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:42.656781 17184 tablet_bootstrap.cc:654] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.664135 16811 catalog_manager.cc:5697] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:42.649587 17379 leader_election.cc:290] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:42.675081 16813 catalog_manager.cc:5697] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:42.681457 17104 leader_election.cc:304] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.682446 17452 raft_consensus.cc:2804] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.683233 17452 raft_consensus.cc:697] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:42.686460 17452 consensus_queue.cc:237] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.693186 17186 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:42.693804 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:42.694857 16989 raft_consensus.cc:3060] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:42.696262 17031 leader_election.cc:343] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.693851 17186 ts_tablet_manager.cc:1403] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.211s	user 0.014s	sys 0.006s
I20260709 13:05:42.699605 17330 raft_consensus.cc:493] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.700210 17330 raft_consensus.cc:515] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.702421 17459 consensus_queue.cc:1048] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:42.718976 17185 tablet_bootstrap.cc:654] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.713290 17062 raft_consensus.cc:2468] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:42.732817 17183 tablet_bootstrap.cc:654] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.728533 17279 raft_consensus.cc:493] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.738034 17279 raft_consensus.cc:515] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:42.741789 17186 raft_consensus.cc:359] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.744282 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:42.744611 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:42.746053 17030 leader_election.cc:343] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.746680 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:42.743363 17186 raft_consensus.cc:385] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.747972 17255 consensus_queue.cc:1048] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:42.748008 17186 raft_consensus.cc:740] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
W20260709 13:05:42.748155 17028 leader_election.cc:343] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.749264 17028 leader_election.cc:304] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:42.750587 17379 raft_consensus.cc:2749] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:05:42.747551 16954 leader_election.cc:343] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.749760 17186 consensus_queue.cc:260] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:42.781737 17184 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:42.782430 17184 ts_tablet_manager.cc:1403] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.220s	user 0.027s	sys 0.027s
I20260709 13:05:42.781800 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:42.785578 16913 raft_consensus.cc:2468] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:42.787969 16957 leader_election.cc:304] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:42.788766 17459 raft_consensus.cc:2804] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.788724 16989 raft_consensus.cc:2468] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:42.789436 17459 raft_consensus.cc:493] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.789870 17459 raft_consensus.cc:3060] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.790577 17030 leader_election.cc:304] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.743436 17279 leader_election.cc:290] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:42.798192 17186 ts_tablet_manager.cc:1434] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.099s	user 0.009s	sys 0.000s
I20260709 13:05:42.797029 17428 raft_consensus.cc:2804] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.799767 17454 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.800275 17454 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
W20260709 13:05:42.802240 16954 consensus_peers.cc:597] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b -> Peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Couldn't send request to peer 40c8156681dd4a7bb2d037afbeb01580. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:42.802548 17454 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:42.808382 17428 raft_consensus.cc:697] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:42.793067 17330 leader_election.cc:290] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:42.816412 17428 consensus_queue.cc:237] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.810878 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:42.812842 17379 consensus_queue.cc:1048] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:05:42.823758 17105 leader_election.cc:343] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.826359 17184 raft_consensus.cc:359] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.827191 17184 raft_consensus.cc:385] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.827461 17184 raft_consensus.cc:740] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.828320 17184 consensus_queue.cc:260] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.830979 17186 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:42.836129 16814 catalog_manager.cc:5697] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:42.842379 17330 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:42.843192 17330 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.859087 17185 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:42.860061 17185 ts_tablet_manager.cc:1403] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.219s	user 0.012s	sys 0.012s
I20260709 13:05:42.863319 17185 raft_consensus.cc:359] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.864596 17185 raft_consensus.cc:385] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.864939 17185 raft_consensus.cc:740] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.853259 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:42.865746 17185 consensus_queue.cc:260] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:42.853375 17330 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:42.867182 17459 raft_consensus.cc:515] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.865940 16989 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:42.870893 17185 ts_tablet_manager.cc:1434] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:05:42.872654 17185 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:42.876274 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:42.882441 16957 leader_election.cc:343] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.888267 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:42.889034 17137 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
W20260709 13:05:42.893378 17031 consensus_peers.cc:597] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:42.898406 17459 leader_election.cc:290] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:42.900082 17186 tablet_bootstrap.cc:654] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.900660 17104 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.902508 17454 raft_consensus.cc:2804] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.902897 17454 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.903268 17454 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.904119 16954 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:42.905711 17330 raft_consensus.cc:2804] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:42.906159 17330 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:42.906544 17330 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.884697 16813 catalog_manager.cc:5697] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:42.911630 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:42.912613 16912 raft_consensus.cc:3060] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:42.919152 17330 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.922566 17184 ts_tablet_manager.cc:1434] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.140s	user 0.006s	sys 0.000s
I20260709 13:05:42.923754 17184 tablet_bootstrap.cc:492] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:42.870669 16811 catalog_manager.cc:5697] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:42.933698 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:42.935179 16912 raft_consensus.cc:2468] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:42.934233 17183 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:42.937343 16957 leader_election.cc:304] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:42.937752 17183 ts_tablet_manager.cc:1403] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.326s	user 0.014s	sys 0.065s
I20260709 13:05:42.938750 17459 raft_consensus.cc:2804] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:42.939272 17459 raft_consensus.cc:697] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:42.940099 17459 consensus_queue.cc:237] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.940958 17183 raft_consensus.cc:359] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.942992 17183 raft_consensus.cc:385] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:42.943640 17183 raft_consensus.cc:740] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:42.954344 17183 consensus_queue.cc:260] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:42.957468 17183 ts_tablet_manager.cc:1434] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.005s
I20260709 13:05:42.951404 17185 tablet_bootstrap.cc:654] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:42.943218 16954 consensus_peers.cc:597] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Couldn't send request to peer 40c8156681dd4a7bb2d037afbeb01580. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:42.965652 17183 tablet_bootstrap.cc:492] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
W20260709 13:05:42.965883 16954 leader_election.cc:343] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.972723 17183 tablet_bootstrap.cc:654] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:42.977221 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:42.960314 17330 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:42.945225 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:42.979190 16957 leader_election.cc:343] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:42.981369 16989 raft_consensus.cc:1275] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:05:42.982883 17451 consensus_queue.cc:1048] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:42.995689 17330 consensus_queue.cc:1048] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:43.014420 17352 raft_consensus.cc:493] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.018141 17352 raft_consensus.cc:515] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.019012 17183 tablet_bootstrap.cc:492] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:43.019757 17183 ts_tablet_manager.cc:1403] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.054s	user 0.010s	sys 0.007s
I20260709 13:05:43.023474 17183 raft_consensus.cc:359] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.024417 17183 raft_consensus.cc:385] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.024755 17183 raft_consensus.cc:740] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.029745 17183 consensus_queue.cc:260] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.036466 17183 ts_tablet_manager.cc:1434] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.002s
I20260709 13:05:43.034469 17428 raft_consensus.cc:493] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.053433 17183 tablet_bootstrap.cc:492] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:43.057602 17184 tablet_bootstrap.cc:654] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.057717 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:43.058693 16988 raft_consensus.cc:2393] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:43.061138 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:43.063598 16879 leader_election.cc:304] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:43.069667 17421 raft_consensus.cc:3060] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.078372 17186 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:43.078969 17186 ts_tablet_manager.cc:1403] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.280s	user 0.014s	sys 0.011s
I20260709 13:05:43.084175 17186 raft_consensus.cc:359] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.084882 17186 raft_consensus.cc:385] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.085162 17186 raft_consensus.cc:740] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.085817 17186 consensus_queue.cc:260] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.086524 17063 raft_consensus.cc:1275] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:05:43.086761 17279 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.087309 17279 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.088019 17186 ts_tablet_manager.cc:1434] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.006s
I20260709 13:05:43.088510 17451 consensus_queue.cc:1048] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:05:43.091928 16879 consensus_peers.cc:597] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:43.057847 17352 leader_election.cc:290] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:43.104769 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:43.105989 17454 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:43.066841 17428 raft_consensus.cc:515] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.114024 17137 raft_consensus.cc:2393] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:43.130479 16954 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:43.134361 17186 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:43.132863 17330 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 13:05:43.148245 17031 leader_election.cc:343] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.149088 17279 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:43.159922 17454 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:43.160841 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:43.161921 17428 leader_election.cc:290] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:43.168226 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:43.170308 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:43.171835 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:43.178721 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:43.179376 17452 consensus_queue.cc:1048] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:43.180728 17418 consensus_queue.cc:1048] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:43.216598 16990 raft_consensus.cc:2393] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:43.218920 17183 tablet_bootstrap.cc:654] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.225943 16814 catalog_manager.cc:5697] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:43.227600 17421 raft_consensus.cc:2749] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:05:43.230161 17031 leader_election.cc:343] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:43.230419 17030 leader_election.cc:343] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.245528 17030 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:43.252880 17379 raft_consensus.cc:2749] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.254426 17428 consensus_queue.cc:1048] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 13:05:43.269272 17184 tablet_bootstrap.cc:492] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:43.269680 17185 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:43.269937 17184 ts_tablet_manager.cc:1403] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.346s	user 0.025s	sys 0.029s
I20260709 13:05:43.270262 17185 ts_tablet_manager.cc:1403] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.398s	user 0.038s	sys 0.104s
I20260709 13:05:43.261879 17455 consensus_queue.cc:1048] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:43.274091 17186 tablet_bootstrap.cc:654] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:43.264866 17028 leader_election.cc:343] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.275131 17028 leader_election.cc:304] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:43.251888 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 66357.9us
I20260709 13:05:43.276623 17428 raft_consensus.cc:2749] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:05:43.276824 17105 leader_election.cc:343] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.277359 17105 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:43.279739 17454 raft_consensus.cc:493] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.280550 17454 raft_consensus.cc:515] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:43.284330 17454 leader_election.cc:290] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:43.272969 17184 raft_consensus.cc:359] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.290621 17184 raft_consensus.cc:385] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.292078 17184 raft_consensus.cc:740] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.294890 17185 raft_consensus.cc:359] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.295967 17185 raft_consensus.cc:385] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.289991 17454 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.308161 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:43.293390 17184 consensus_queue.cc:260] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.296226 17185 raft_consensus.cc:740] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
W20260709 13:05:43.313100 17030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 61796.9us
I20260709 13:05:43.321218 16912 raft_consensus.cc:2393] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:43.323041 17186 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:43.323604 17186 ts_tablet_manager.cc:1403] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.192s	user 0.010s	sys 0.007s
I20260709 13:05:43.317773 17185 consensus_queue.cc:260] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.330591 17186 raft_consensus.cc:359] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.331327 17186 raft_consensus.cc:385] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.331593 17186 raft_consensus.cc:740] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.332248 17186 consensus_queue.cc:260] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.332535 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:43.333654 17428 consensus_queue.cc:1048] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:43.340778 17186 ts_tablet_manager.cc:1434] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:05:43.342180 17186 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:43.347147 17184 ts_tablet_manager.cc:1434] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.077s	user 0.007s	sys 0.002s
I20260709 13:05:43.349221 17184 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:43.355047 17185 ts_tablet_manager.cc:1434] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.084s	user 0.009s	sys 0.000s
I20260709 13:05:43.344729 17497 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.357537 17497 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:43.356434 17105 consensus_peers.cc:597] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:43.365623 17185 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:43.386718 17497 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:43.392344 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:43.394007 17452 consensus_queue.cc:1048] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:43.395211 17137 raft_consensus.cc:3060] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.396600 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:43.397234 17186 tablet_bootstrap.cc:654] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.403170 17185 tablet_bootstrap.cc:654] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.410987 17028 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:43.425760 17184 tablet_bootstrap.cc:654] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.424012 17516 raft_consensus.cc:2749] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.442227 17183 tablet_bootstrap.cc:492] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:43.443135 17183 ts_tablet_manager.cc:1403] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.390s	user 0.025s	sys 0.006s
I20260709 13:05:43.449287 17105 leader_election.cc:304] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:43.444629 17459 raft_consensus.cc:493] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:05:43.452961 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 79506.4us
I20260709 13:05:43.466611 17352 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.467186 17352 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.470197 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:43.454927 17512 raft_consensus.cc:3060] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.480684 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:43.481895 17185 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:43.483415 17185 ts_tablet_manager.cc:1403] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.127s	user 0.017s	sys 0.010s
I20260709 13:05:43.484050 16879 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:43.485934 17421 raft_consensus.cc:2749] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.454103 17459 raft_consensus.cc:515] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.484110 17183 raft_consensus.cc:359] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.488305 17183 raft_consensus.cc:385] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.488605 17183 raft_consensus.cc:740] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.488942 17459 leader_election.cc:290] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:43.489310 17183 consensus_queue.cc:260] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.498199 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:43.498322 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:43.503430 17063 raft_consensus.cc:2468] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:43.469298 17352 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:43.506197 17183 ts_tablet_manager.cc:1434] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.063s	user 0.007s	sys 0.001s
I20260709 13:05:43.507707 17183 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
W20260709 13:05:43.504940 16954 leader_election.cc:343] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.508354 17470 consensus_queue.cc:1048] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:43.508589 16954 leader_election.cc:304] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:43.509997 17471 raft_consensus.cc:2804] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:43.510382 17471 raft_consensus.cc:493] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:43.510732 17471 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.540315 17454 raft_consensus.cc:493] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:43.540988 17454 raft_consensus.cc:515] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.545948 17183 tablet_bootstrap.cc:654] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.545150 17454 leader_election.cc:290] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:43.546947 17279 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.547953 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:43.551996 17279 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.555222 17471 raft_consensus.cc:515] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.555627 17279 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:43.558674 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:43.559283 17064 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.560643 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:43.562032 16954 leader_election.cc:343] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.567205 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:43.568871 17186 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
W20260709 13:05:43.571414 17028 leader_election.cc:343] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.574625 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:43.575752 17185 raft_consensus.cc:359] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:43.576411 17031 leader_election.cc:343] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.577006 17031 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:43.580698 17279 raft_consensus.cc:2749] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.571702 17186 ts_tablet_manager.cc:1403] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.230s	user 0.023s	sys 0.014s
I20260709 13:05:43.569988 17512 raft_consensus.cc:2749] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.589692 17186 raft_consensus.cc:359] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.593744 17186 raft_consensus.cc:385] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.594198 17186 raft_consensus.cc:740] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.576432 17185 raft_consensus.cc:385] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.595203 17185 raft_consensus.cc:740] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.596024 17185 consensus_queue.cc:260] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.546558 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:43.544382 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 47554us
I20260709 13:05:43.599746 17470 consensus_queue.cc:1048] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:43.600193 17104 leader_election.cc:304] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:43.601778 17454 raft_consensus.cc:2749] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:43.602460 17065 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.629592 17185 ts_tablet_manager.cc:1434] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.146s	user 0.007s	sys 0.001s
I20260709 13:05:43.630743 17185 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:43.632360 17471 leader_election.cc:290] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:43.597604 17186 consensus_queue.cc:260] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.633721 17064 raft_consensus.cc:2468] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:43.635321 16954 leader_election.cc:304] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:43.636713 17470 raft_consensus.cc:2804] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:43.637241 17470 raft_consensus.cc:697] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:43.638128 17470 consensus_queue.cc:237] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.641927 17183 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:43.642490 17183 ts_tablet_manager.cc:1403] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.135s	user 0.018s	sys 0.025s
I20260709 13:05:43.635345 17186 ts_tablet_manager.cc:1434] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.049s	user 0.005s	sys 0.001s
I20260709 13:05:43.645133 17186 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:43.650660 16915 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.652761 17184 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:43.653685 17184 ts_tablet_manager.cc:1403] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.306s	user 0.010s	sys 0.008s
I20260709 13:05:43.647169 17183 raft_consensus.cc:359] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.658862 17183 raft_consensus.cc:385] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.659904 17183 raft_consensus.cc:740] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.662477 17183 consensus_queue.cc:260] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.671262 17183 ts_tablet_manager.cc:1434] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.002s
I20260709 13:05:43.672418 17183 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:43.701088 17184 raft_consensus.cc:359] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:43.704295 17184 raft_consensus.cc:385] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.704528 17184 raft_consensus.cc:740] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.706264 17184 consensus_queue.cc:260] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:43.711851 17184 ts_tablet_manager.cc:1434] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.058s	user 0.006s	sys 0.000s
I20260709 13:05:43.714452 16811 catalog_manager.cc:5697] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:43.719038 17184 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:43.728732 17184 tablet_bootstrap.cc:654] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.733783 17185 tablet_bootstrap.cc:654] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.755785 17186 tablet_bootstrap.cc:654] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.757296 17184 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:43.757836 17185 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:43.758026 17184 ts_tablet_manager.cc:1403] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.040s	user 0.016s	sys 0.013s
I20260709 13:05:43.758594 17183 tablet_bootstrap.cc:654] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.760843 17185 ts_tablet_manager.cc:1403] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.130s	user 0.012s	sys 0.001s
I20260709 13:05:43.762919 17514 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.763118 17184 raft_consensus.cc:359] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.763522 17514 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.763993 17184 raft_consensus.cc:385] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.765969 17184 raft_consensus.cc:740] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.767678 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:43.768199 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:43.765991 17514 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:43.768249 17184 consensus_queue.cc:260] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.768424 17063 raft_consensus.cc:2468] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:43.770587 16879 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:43.774189 17514 raft_consensus.cc:2804] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:43.776803 17185 raft_consensus.cc:359] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.777549 17185 raft_consensus.cc:385] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.777796 17185 raft_consensus.cc:740] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.778390 17185 consensus_queue.cc:260] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.775075 17514 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:05:43.779201 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 13:05:43.775179 16879 leader_election.cc:343] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.780424 17514 raft_consensus.cc:3060] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.784271 17186 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:43.784794 17186 ts_tablet_manager.cc:1403] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.140s	user 0.010s	sys 0.007s
I20260709 13:05:43.787657 17186 raft_consensus.cc:359] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.788352 17186 raft_consensus.cc:385] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.788642 17186 raft_consensus.cc:740] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.789235 17186 consensus_queue.cc:260] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.790144 17185 ts_tablet_manager.cc:1434] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 13:05:43.792055 17186 ts_tablet_manager.cc:1434] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 13:05:43.790385 17184 ts_tablet_manager.cc:1434] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260709 13:05:43.793356 17184 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:43.798422 17185 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:43.798316 17186 tablet_bootstrap.cc:492] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:43.801651 17514 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.805295 17183 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:43.806078 17183 ts_tablet_manager.cc:1403] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.134s	user 0.008s	sys 0.009s
I20260709 13:05:43.809947 17186 tablet_bootstrap.cc:654] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.824148 17184 tablet_bootstrap.cc:654] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.827574 17183 raft_consensus.cc:359] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.828627 17183 raft_consensus.cc:385] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.828907 17183 raft_consensus.cc:740] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.829594 17183 consensus_queue.cc:260] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.832438 17183 ts_tablet_manager.cc:1434] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.001s
I20260709 13:05:43.833495 17514 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:43.834136 17186 tablet_bootstrap.cc:492] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:43.834630 17186 ts_tablet_manager.cc:1403] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.015s
I20260709 13:05:43.834687 17185 tablet_bootstrap.cc:654] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.835216 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:43.835718 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:43.835886 17135 raft_consensus.cc:3060] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.836413 17065 raft_consensus.cc:3060] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.837817 17186 raft_consensus.cc:359] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.838544 17186 raft_consensus.cc:385] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.838804 17186 raft_consensus.cc:740] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.839995 17514 raft_consensus.cc:493] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.840950 17186 consensus_queue.cc:260] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.843209 17186 ts_tablet_manager.cc:1434] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:05:43.844133 17186 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:43.849509 17516 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.849809 17139 raft_consensus.cc:3060] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:43.850417 17186 tablet_bootstrap.cc:654] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.850390 17516 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:43.840689 17514 raft_consensus.cc:515] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.862134 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:43.863147 17186 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
W20260709 13:05:43.863436 16881 leader_election.cc:343] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.863679 17186 ts_tablet_manager.cc:1403] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.003s
I20260709 13:05:43.865592 17065 raft_consensus.cc:2468] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:43.865885 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:43.861374 17514 leader_election.cc:290] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:43.866487 17186 raft_consensus.cc:359] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.866900 16879 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:43.867287 17186 raft_consensus.cc:385] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.867609 17186 raft_consensus.cc:740] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.867955 17514 raft_consensus.cc:2804] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:43.868256 17186 consensus_queue.cc:260] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.870718 17186 ts_tablet_manager.cc:1434] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:05:43.874989 17186 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:43.874809 17183 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:43.869344 17514 raft_consensus.cc:697] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:43.870326 17063 raft_consensus.cc:2468] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 0.
I20260709 13:05:43.870726 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:43.886924 17030 leader_election.cc:343] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.890090 17183 tablet_bootstrap.cc:654] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.891337 17516 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:43.905844 17184 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:43.906420 17184 ts_tablet_manager.cc:1403] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.113s	user 0.005s	sys 0.007s
I20260709 13:05:43.902344 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:43.917727 17028 leader_election.cc:343] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.918335 17028 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:43.919492 17516 raft_consensus.cc:2749] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.924008 17186 tablet_bootstrap.cc:654] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:43.928021 17279 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:43.928610 17279 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.930755 17279 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:43.938197 16879 leader_election.cc:304] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:43.938606 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:43.900758 17514 consensus_queue.cc:237] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.943107 17421 raft_consensus.cc:2804] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:43.943562 17421 raft_consensus.cc:493] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:43.943960 17421 raft_consensus.cc:3060] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:43.947729 17031 leader_election.cc:343] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.955890 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:43.957696 17028 leader_election.cc:343] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:43.958199 17028 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:43.960340 17516 raft_consensus.cc:2749] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:43.967612 17185 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:43.968214 17185 ts_tablet_manager.cc:1403] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.170s	user 0.027s	sys 0.022s
I20260709 13:05:43.973040 17186 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:43.973982 17186 ts_tablet_manager.cc:1403] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.099s	user 0.015s	sys 0.010s
I20260709 13:05:43.973060 17184 raft_consensus.cc:359] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:43.975304 17184 raft_consensus.cc:385] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.975574 17184 raft_consensus.cc:740] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.977697 17184 consensus_queue.cc:260] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:43.980893 17185 raft_consensus.cc:359] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:43.981685 17185 raft_consensus.cc:385] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.981995 17185 raft_consensus.cc:740] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:43.980407 17421 raft_consensus.cc:515] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:43.984311 17184 ts_tablet_manager.cc:1434] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.078s	user 0.008s	sys 0.000s
I20260709 13:05:43.989863 17184 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:43.992172 17183 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:43.992776 17183 ts_tablet_manager.cc:1403] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.118s	user 0.014s	sys 0.005s
I20260709 13:05:43.998255 17186 raft_consensus.cc:359] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:43.999168 17186 raft_consensus.cc:385] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:43.982875 17185 consensus_queue.cc:260] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.001912 17185 ts_tablet_manager.cc:1434] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.000s
I20260709 13:05:44.002859 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
W20260709 13:05:44.004599 16881 leader_election.cc:343] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.005364 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:44.006049 17065 raft_consensus.cc:3060] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.006700 17185 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:43.999825 17186 raft_consensus.cc:740] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.013899 16809 catalog_manager.cc:5697] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:44.015215 17183 raft_consensus.cc:359] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.015852 17183 raft_consensus.cc:385] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.016187 17183 raft_consensus.cc:740] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.019022 17135 raft_consensus.cc:2468] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:44.022465 17183 consensus_queue.cc:260] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.025846 17183 ts_tablet_manager.cc:1434] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.001s
I20260709 13:05:44.027102 17183 tablet_bootstrap.cc:492] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:44.012482 17186 consensus_queue.cc:260] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:44.023465 17065 raft_consensus.cc:2468] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:44.011404 17421 leader_election.cc:290] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:44.041860 17186 ts_tablet_manager.cc:1434] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.001s
I20260709 13:05:44.043164 17186 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:44.046638 16879 leader_election.cc:304] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:44.053927 17421 raft_consensus.cc:2804] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:44.054512 17421 raft_consensus.cc:697] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:44.062387 17185 tablet_bootstrap.cc:654] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.063629 17184 tablet_bootstrap.cc:654] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.055310 17421 consensus_queue.cc:237] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.067883 17186 tablet_bootstrap.cc:654] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.074020 17183 tablet_bootstrap.cc:654] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.082695 17185 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:44.083014 17186 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:44.087956 17186 ts_tablet_manager.cc:1403] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.045s	user 0.010s	sys 0.004s
W20260709 13:05:44.089831 16954 consensus_peers.cc:597] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Couldn't send request to peer 40c8156681dd4a7bb2d037afbeb01580. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:44.090026 17459 raft_consensus.cc:493] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.090507 17459 raft_consensus.cc:515] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.091230 17186 raft_consensus.cc:359] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.091995 17186 raft_consensus.cc:385] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.092301 17459 leader_election.cc:290] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:44.084667 17183 tablet_bootstrap.cc:492] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:44.093257 17183 ts_tablet_manager.cc:1403] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.066s	user 0.010s	sys 0.003s
I20260709 13:05:44.094089 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:44.096052 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:44.096187 17183 raft_consensus.cc:359] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.096794 17135 raft_consensus.cc:2468] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:44.097126 17183 raft_consensus.cc:385] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.092324 17186 raft_consensus.cc:740] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.097627 17471 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.098351 17183 raft_consensus.cc:740] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.098562 17471 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.099152 17183 consensus_queue.cc:260] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.100827 17471 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:44.101285 16809 catalog_manager.cc:5697] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:44.083105 17184 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:44.103446 17183 ts_tablet_manager.cc:1434] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:05:44.103925 17184 ts_tablet_manager.cc:1403] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.114s	user 0.016s	sys 0.000s
I20260709 13:05:44.100508 17186 consensus_queue.cc:260] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.106963 17183 tablet_bootstrap.cc:492] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:44.107781 17186 ts_tablet_manager.cc:1434] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.001s
I20260709 13:05:44.108805 17186 tablet_bootstrap.cc:492] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b: Bootstrap starting.
I20260709 13:05:44.083405 17185 ts_tablet_manager.cc:1403] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.077s	user 0.014s	sys 0.005s
I20260709 13:05:44.114588 17183 tablet_bootstrap.cc:654] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.114451 17184 raft_consensus.cc:359] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.115327 17184 raft_consensus.cc:385] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.115688 17184 raft_consensus.cc:740] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.102476 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:44.116712 17065 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:44.116530 17184 consensus_queue.cc:260] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.115859 17185 raft_consensus.cc:359] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.118335 17185 raft_consensus.cc:385] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.118687 17185 raft_consensus.cc:740] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.119402 17185 consensus_queue.cc:260] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.124110 17184 ts_tablet_manager.cc:1434] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 13:05:44.125314 17185 ts_tablet_manager.cc:1434] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 13:05:44.126750 17185 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:44.125708 17184 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:44.129240 17186 tablet_bootstrap.cc:654] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.130792 17183 tablet_bootstrap.cc:492] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:44.131431 17183 ts_tablet_manager.cc:1403] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.025s	user 0.019s	sys 0.000s
I20260709 13:05:44.134825 17183 raft_consensus.cc:359] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.135730 17183 raft_consensus.cc:385] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.136086 17183 raft_consensus.cc:740] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.137008 17183 consensus_queue.cc:260] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.139164 17185 tablet_bootstrap.cc:654] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.139886 17183 ts_tablet_manager.cc:1434] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:05:44.141141 17183 tablet_bootstrap.cc:492] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:44.141724 17184 tablet_bootstrap.cc:654] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.143088 17454 raft_consensus.cc:493] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.143639 17454 raft_consensus.cc:515] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:44.144287 17186 tablet_bootstrap.cc:492] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b: No bootstrap required, opened a new log
I20260709 13:05:44.144795 17186 ts_tablet_manager.cc:1403] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.000s
I20260709 13:05:44.147630 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:44.147914 17186 raft_consensus.cc:359] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.148403 17065 raft_consensus.cc:2468] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:44.148646 17186 raft_consensus.cc:385] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.148993 17186 raft_consensus.cc:740] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Initialized, Role: FOLLOWER
W20260709 13:05:44.098237 16957 leader_election.cc:343] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.150118 17102 leader_election.cc:304] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
W20260709 13:05:44.150785 16957 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 52600us
I20260709 13:05:44.151178 17454 leader_election.cc:290] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:44.151885 17454 raft_consensus.cc:2804] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:44.152277 17454 raft_consensus.cc:493] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.152642 17454 raft_consensus.cc:3060] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.154568 17186 consensus_queue.cc:260] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.155133 16954 leader_election.cc:304] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:44.156365 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 54187.8us
I20260709 13:05:44.157625 17471 raft_consensus.cc:2804] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:44.158113 17471 raft_consensus.cc:493] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.158506 17471 raft_consensus.cc:3060] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.162547 17186 ts_tablet_manager.cc:1434] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 13:05:44.165915 17184 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:44.165915 17470 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.166466 17184 ts_tablet_manager.cc:1403] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.041s	user 0.015s	sys 0.000s
I20260709 13:05:44.166788 17470 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.168818 17470 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:44.169206 17514 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.169498 17184 raft_consensus.cc:359] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.170226 17184 raft_consensus.cc:385] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.169876 17514 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.168761 17454 raft_consensus.cc:515] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:44.172698 17428 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.172994 17183 tablet_bootstrap.cc:654] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.173535 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:44.173738 17514 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.173282 17428 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.174407 17135 raft_consensus.cc:2393] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:44.175490 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.175730 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:44.176445 17065 raft_consensus.cc:3060] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.170557 17184 raft_consensus.cc:740] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.176726 17428 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.177639 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.177757 17184 consensus_queue.cc:260] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.178812 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:44.179980 16990 raft_consensus.cc:2468] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:44.182736 17030 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:44.180035 17184 ts_tablet_manager.cc:1434] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 13:05:44.183780 17279 raft_consensus.cc:493] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.184875 17184 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:44.184973 17428 raft_consensus.cc:2804] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:44.189589 17428 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.190122 17428 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.190217 17065 raft_consensus.cc:2468] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:44.191738 17102 leader_election.cc:304] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:44.192854 17452 raft_consensus.cc:2804] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:44.184270 17279 raft_consensus.cc:515] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:44.193701 17452 raft_consensus.cc:697] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:44.194679 17452 consensus_queue.cc:237] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:44.195168 17279 leader_election.cc:290] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:44.176570 16988 raft_consensus.cc:2393] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:44.197669 16881 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:44.186560 17454 leader_election.cc:290] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:44.196534 17184 tablet_bootstrap.cc:654] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:44.201730 17105 leader_election.cc:343] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.203034 17418 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.208115 17183 tablet_bootstrap.cc:492] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:44.208681 17183 ts_tablet_manager.cc:1403] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.068s	user 0.005s	sys 0.008s
I20260709 13:05:44.210618 17418 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:44.210963 16954 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:44.230095 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:44.233688 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:44.237483 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:44.239668 17470 raft_consensus.cc:2804] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:44.240180 17470 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.240526 17470 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.243929 17184 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:44.244745 17184 ts_tablet_manager.cc:1403] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.060s	user 0.013s	sys 0.001s
I20260709 13:05:44.247318 17470 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.249550 17470 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:44.256527 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:44.213326 17183 raft_consensus.cc:359] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.260962 17183 raft_consensus.cc:385] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.261301 17183 raft_consensus.cc:740] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.294164 17183 consensus_queue.cc:260] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.296123 17183 ts_tablet_manager.cc:1434] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.086s	user 0.014s	sys 0.007s
I20260709 13:05:44.297215 17183 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:44.221462 17452 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.298368 17452 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.299507 17135 raft_consensus.cc:2468] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 0.
I20260709 13:05:44.299520 17514 raft_consensus.cc:493] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.300408 17514 raft_consensus.cc:515] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.302604 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:44.303267 17065 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.306141 17514 leader_election.cc:290] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.306618 17452 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.310177 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:44.313766 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:44.314987 17459 consensus_queue.cc:1048] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:44.315371 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:44.317111 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
W20260709 13:05:44.319798 17031 leader_election.cc:343] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.323215 17183 tablet_bootstrap.cc:654] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.330756 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.334708 17516 raft_consensus.cc:493] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.335319 17516 raft_consensus.cc:515] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.338097 17065 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:44.343669 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.345155 17184 raft_consensus.cc:359] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.346336 17184 raft_consensus.cc:385] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.346771 17184 raft_consensus.cc:740] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.348157 17516 leader_election.cc:290] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:44.347756 17184 consensus_queue.cc:260] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.348786 17516 raft_consensus.cc:493] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.252905 17471 raft_consensus.cc:515] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.349573 17516 raft_consensus.cc:515] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:44.345641 17030 leader_election.cc:343] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.351490 17471 leader_election.cc:290] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:44.352883 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.353664 17184 ts_tablet_manager.cc:1434] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.108s	user 0.010s	sys 0.004s
I20260709 13:05:44.353842 16988 raft_consensus.cc:2468] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:44.354936 16879 leader_election.cc:304] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:44.356513 17516 leader_election.cc:290] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.359508 17452 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.359311 17428 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.364396 17184 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:44.368633 17030 leader_election.cc:304] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
W20260709 13:05:44.370836 17105 leader_election.cc:343] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.375164 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:44.376717 17563 raft_consensus.cc:493] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.370448 16954 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:44.378535 17255 raft_consensus.cc:2804] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:44.379007 17255 raft_consensus.cc:697] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:44.379755 17255 consensus_queue.cc:237] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.381336 17514 raft_consensus.cc:2749] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:44.388933 17279 raft_consensus.cc:2804] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:44.389389 17279 raft_consensus.cc:493] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.389750 17279 raft_consensus.cc:3060] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.391505 17183 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:44.392398 17183 ts_tablet_manager.cc:1403] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.095s	user 0.015s	sys 0.000s
I20260709 13:05:44.377499 17563 raft_consensus.cc:515] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:44.378664 17031 leader_election.cc:343] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.395859 17563 leader_election.cc:290] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:44.377545 17452 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.399344 17452 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
W20260709 13:05:44.387374 16957 leader_election.cc:343] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.407836 17279 raft_consensus.cc:515] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.392478 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:44.410207 16957 leader_election.cc:343] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.412061 17184 tablet_bootstrap.cc:654] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:44.414122 16957 leader_election.cc:343] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:44.425679 17105 leader_election.cc:343] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.445935 17183 raft_consensus.cc:359] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.448866 17183 raft_consensus.cc:385] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.449330 17183 raft_consensus.cc:740] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.450286 17183 consensus_queue.cc:260] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.458174 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:44.459223 17064 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:44.443006 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:44.465617 17030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 97075.2us
I20260709 13:05:44.464728 16954 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:44.452553 16957 leader_election.cc:343] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.462666 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:44.473793 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:44.489554 17063 raft_consensus.cc:2393] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate 71a10d4adec64d58ba645d68425ed531 in this term.
W20260709 13:05:44.492575 17031 leader_election.cc:343] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.495314 16910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:44.498018 17185 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:44.498643 17185 ts_tablet_manager.cc:1403] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.372s	user 0.075s	sys 0.085s
I20260709 13:05:44.498754 17454 raft_consensus.cc:493] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.499284 17428 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.480432 17563 raft_consensus.cc:2804] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:44.522534 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:44.522689 17185 raft_consensus.cc:359] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.523414 17185 raft_consensus.cc:385] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.523736 17185 raft_consensus.cc:740] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.524411 17185 consensus_queue.cc:260] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.489845 17135 raft_consensus.cc:3060] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.512444 17563 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.523744 17279 leader_election.cc:290] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.540333 17185 ts_tablet_manager.cc:1434] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.001s
I20260709 13:05:44.499303 17454 raft_consensus.cc:515] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.536314 17563 raft_consensus.cc:3060] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.549579 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:44.552078 17031 leader_election.cc:343] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.559643 17102 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:44.553346 16915 raft_consensus.cc:3060] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.561993 17136 raft_consensus.cc:2468] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:44.515091 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:44.565246 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:44.577430 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.577874 16910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:44.578469 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.583123 17183 ts_tablet_manager.cc:1434] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.190s	user 0.013s	sys 0.023s
I20260709 13:05:44.579006 17184 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:44.584208 17183 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:44.584406 17184 ts_tablet_manager.cc:1403] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.222s	user 0.015s	sys 0.005s
I20260709 13:05:44.581089 17454 leader_election.cc:290] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:44.572933 17558 raft_consensus.cc:2804] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:05:44.585852 17028 leader_election.cc:343] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.593888 17028 leader_election.cc:304] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:44.595382 17575 raft_consensus.cc:493] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.595468 17279 raft_consensus.cc:2749] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:44.580911 17421 consensus_queue.cc:1048] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 13:05:44.591302 17558 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.589236 16987 raft_consensus.cc:3060] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:44.583348 17031 leader_election.cc:343] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.592756 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:44.596025 17575 raft_consensus.cc:515] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.615736 17184 raft_consensus.cc:359] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.622915 17184 raft_consensus.cc:385] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:44.623328 17184 raft_consensus.cc:740] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:44.627120 17184 consensus_queue.cc:260] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.604822 17558 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.641634 17184 ts_tablet_manager.cc:1434] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.057s	user 0.006s	sys 0.004s
I20260709 13:05:44.635731 16988 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.624614 17028 leader_election.cc:304] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:44.646584 17183 tablet_bootstrap.cc:654] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.647214 17577 raft_consensus.cc:2804] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:44.647647 17577 raft_consensus.cc:493] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:44.647958 17577 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.612465 17135 raft_consensus.cc:2468] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:44.651065 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:44.654515 17104 leader_election.cc:343] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.654943 17184 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
W20260709 13:05:44.639156 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 63368us
I20260709 13:05:44.651089 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.671429 16954 leader_election.cc:304] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:44.673138 17185 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:44.685834 17564 raft_consensus.cc:2804] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:44.686494 17564 raft_consensus.cc:697] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:44.687317 17564 consensus_queue.cc:237] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.689584 16987 raft_consensus.cc:2468] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:44.693085 17030 leader_election.cc:304] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:44.698287 16987 raft_consensus.cc:3060] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.700438 17428 raft_consensus.cc:2804] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:05:44.721012 17580 log.cc:927] Time spent T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 13:05:44.722398 17138 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.726730 17563 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.727325 17577 raft_consensus.cc:515] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.728138 16986 raft_consensus.cc:2393] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:44.728976 17060 raft_consensus.cc:2393] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:44.738525 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:44.700066 16957 leader_election.cc:304] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:44.731600 17575 leader_election.cc:290] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:44.729353 17563 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:44.735586 17428 raft_consensus.cc:697] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
W20260709 13:05:44.746733 17030 leader_election.cc:343] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.749619 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:44.761003 17581 consensus_queue.cc:1048] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260709 13:05:44.760810 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:44.751704 17563 raft_consensus.cc:2749] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:44.750119 17421 consensus_queue.cc:1048] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 13:05:44.750176 17577 leader_election.cc:290] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:44.765312 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:44.760856 16957 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 65889.4us
I20260709 13:05:44.775995 17131 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:44.782467 17104 leader_election.cc:343] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:44.783196 17104 leader_election.cc:304] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:44.784857 16988 raft_consensus.cc:2468] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:44.785660 17104 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:44.787492 17185 tablet_bootstrap.cc:654] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:44.788041 17030 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:44.788947 17516 raft_consensus.cc:2804] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:44.789520 17516 raft_consensus.cc:697] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:44.790447 17516 consensus_queue.cc:237] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.766856 17135 raft_consensus.cc:3060] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:44.787083 17428 consensus_queue.cc:237] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:44.765836 17558 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:44.830938 17558 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:44.886205 16910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
W20260709 13:05:44.888144 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 137932us
W20260709 13:05:44.889336 17105 consensus_peers.cc:597] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:44.899694 17352 consensus_queue.cc:1048] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:05:44.904284 17421 raft_consensus.cc:493] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:05:44.925243 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 120859us
I20260709 13:05:44.938791 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:44.941843 17454 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:44.942415 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:44.943041 17454 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:44.945513 17452 consensus_queue.cc:1048] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:44.953730 17454 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:44.959789 16811 catalog_manager.cc:5697] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:44.963487 17421 raft_consensus.cc:515] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.002871 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:45.004384 17060 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:45.007659 17183 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:45.009112 17183 ts_tablet_manager.cc:1403] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.425s	user 0.018s	sys 0.005s
I20260709 13:05:45.016978 17185 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:45.017879 17568 raft_consensus.cc:493] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:45.018220 17185 ts_tablet_manager.cc:1403] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.345s	user 0.011s	sys 0.009s
I20260709 13:05:45.018858 17568 raft_consensus.cc:515] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.023384 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:45.024451 16914 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:45.039186 17184 tablet_bootstrap.cc:654] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:45.031236 17470 raft_consensus.cc:493] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:45.041800 17421 leader_election.cc:290] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:45.051040 17470 raft_consensus.cc:515] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.030434 17185 raft_consensus.cc:359] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:45.059357 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:45.066839 16988 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
W20260709 13:05:45.083148 17104 leader_election.cc:343] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:45.084756 17104 leader_election.cc:304] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:45.058452 17185 raft_consensus.cc:385] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:45.098207 17185 raft_consensus.cc:740] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:45.111840 17105 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:45.124529 17135 raft_consensus.cc:2468] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:45.125842 17470 leader_election.cc:290] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:45.114252 17185 consensus_queue.cc:260] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:45.133836 17565 raft_consensus.cc:2749] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:45.117121 17568 leader_election.cc:290] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
W20260709 13:05:45.147643 16957 leader_election.cc:343] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:45.157675 17574 raft_consensus.cc:2749] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:45.160730 17183 raft_consensus.cc:359] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.138074 17452 raft_consensus.cc:2804] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:45.167940 17183 raft_consensus.cc:385] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:45.173084 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:45.176517 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:45.173774 17104 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 89284us
I20260709 13:05:45.171593 17183 raft_consensus.cc:740] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:45.171634 17452 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:45.202090 17452 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:45.200982 17183 consensus_queue.cc:260] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.203985 17183 ts_tablet_manager.cc:1434] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.194s	user 0.012s	sys 0.002s
W20260709 13:05:45.212389 17105 leader_election.cc:343] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:45.226527 17183 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:45.234915 17185 ts_tablet_manager.cc:1434] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.216s	user 0.008s	sys 0.002s
I20260709 13:05:45.236500 17185 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:45.269033 17131 raft_consensus.cc:2468] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:45.276093 17575 raft_consensus.cc:2749] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:05:45.296167 17031 consensus_peers.cc:597] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:45.312705 17185 tablet_bootstrap.cc:654] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:45.317505 17183 tablet_bootstrap.cc:654] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:45.334781 17568 raft_consensus.cc:493] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:45.335577 17568 raft_consensus.cc:515] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.337050 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:45.352265 16916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fcfa9f9a5224b69b785589ce292ca3e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:45.353813 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:45.368005 16957 leader_election.cc:343] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:45.369104 17568 leader_election.cc:290] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:45.372829 17060 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:45.375795 17028 leader_election.cc:304] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:45.378052 17184 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:45.378778 17581 raft_consensus.cc:493] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:45.378793 17184 ts_tablet_manager.cc:1403] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.736s	user 0.025s	sys 0.013s
I20260709 13:05:45.379371 17581 raft_consensus.cc:515] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:45.381640 17571 raft_consensus.cc:2804] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:45.382220 17571 raft_consensus.cc:697] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:45.382690 17184 raft_consensus.cc:359] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.383769 17184 raft_consensus.cc:385] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:45.383391 17571 consensus_queue.cc:237] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:45.392432 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:45.396129 17132 raft_consensus.cc:2393] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:45.397707 17581 leader_election.cc:290] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:45.384145 17184 raft_consensus.cc:740] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
W20260709 13:05:45.398298 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 399750us
I20260709 13:05:45.340359 16986 raft_consensus.cc:2468] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
W20260709 13:05:45.416111 17105 leader_election.cc:343] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:45.417205 16879 leader_election.cc:304] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:45.418937 17418 raft_consensus.cc:3060] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:45.421667 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fcfa9f9a5224b69b785589ce292ca3e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:45.422168 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:45.425658 17572 consensus_queue.cc:1048] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:45.427124 17452 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.438674 17184 consensus_queue.cc:260] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.443444 16991 raft_consensus.cc:2468] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
W20260709 13:05:45.452507 17031 consensus_peers.cc:597] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:45.456888 17452 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:45.458709 17104 leader_election.cc:304] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:45.462479 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:45.472198 17104 leader_election.cc:304] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:45.474066 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:45.474548 16914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:45.476158 17418 raft_consensus.cc:2749] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:05:45.477901 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:45.478713 17183 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:45.491783 17183 ts_tablet_manager.cc:1403] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.266s	user 0.021s	sys 0.010s
I20260709 13:05:45.482623 17184 ts_tablet_manager.cc:1434] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.103s	user 0.005s	sys 0.003s
I20260709 13:05:45.483026 17565 raft_consensus.cc:2804] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:45.489666 17570 raft_consensus.cc:2804] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:45.464737 17572 consensus_queue.cc:1048] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260709 13:05:45.493149 17565 raft_consensus.cc:493] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:45.498819 17565 raft_consensus.cc:3060] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:45.491220 16988 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:45.494369 16914 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:45.509864 17570 raft_consensus.cc:493] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:05:45.573707 17593 log.cc:927] Time spent T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:05:45.568635 17570 raft_consensus.cc:3060] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:45.589069 17183 raft_consensus.cc:359] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:45.590708 17183 raft_consensus.cc:385] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:45.590675 17102 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:45.597131 17185 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:45.606664 17185 ts_tablet_manager.cc:1403] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.371s	user 0.018s	sys 0.012s
W20260709 13:05:45.567950 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 67132.3us
I20260709 13:05:45.592396 17183 raft_consensus.cc:740] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
W20260709 13:05:45.597250 17613 log.cc:927] Time spent T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b: Append to log took a long time: real 0.088s	user 0.000s	sys 0.002s
I20260709 13:05:45.616434 17558 raft_consensus.cc:2804] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:45.620479 17558 raft_consensus.cc:697] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:45.621618 17558 consensus_queue.cc:237] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:45.647531 17102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 209555us
I20260709 13:05:45.663264 17566 consensus_queue.cc:1048] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20260709 13:05:45.630168 17183 consensus_queue.cc:260] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:45.697907 16957 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 60675.6us
I20260709 13:05:45.700928 17184 tablet_bootstrap.cc:492] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
W20260709 13:05:45.701941 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 131137us
I20260709 13:05:45.703013 16988 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
W20260709 13:05:45.733057 17607 log.cc:927] Time spent T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b: Append to log took a long time: real 0.196s	user 0.001s	sys 0.005s
I20260709 13:05:45.703296 16916 raft_consensus.cc:3060] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:45.739760 17104 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:45.759156 17185 raft_consensus.cc:359] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:45.763715 17185 raft_consensus.cc:385] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:45.764051 17185 raft_consensus.cc:740] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:45.770665 17183 ts_tablet_manager.cc:1434] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.279s	user 0.009s	sys 0.008s
W20260709 13:05:45.777642 17102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 80688.8us
I20260709 13:05:45.778122 17185 consensus_queue.cc:260] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:45.799126 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:45.807600 17565 raft_consensus.cc:515] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.813695 17131 raft_consensus.cc:2393] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:45.816987 16954 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:45.819242 16957 consensus_peers.cc:597] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:45.839735 17183 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580: Bootstrap starting.
I20260709 13:05:45.825923 17454 raft_consensus.cc:2804] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:45.857864 16914 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:45.861599 17471 raft_consensus.cc:2804] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:45.869884 17471 raft_consensus.cc:697] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:45.888339 17454 raft_consensus.cc:697] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:45.918344 17454 consensus_queue.cc:237] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.907075 17185 ts_tablet_manager.cc:1434] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.300s	user 0.006s	sys 0.010s
I20260709 13:05:45.869262 17570 raft_consensus.cc:515] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:45.935527 16909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:45.967549 17183 tablet_bootstrap.cc:654] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:45.970315 17105 leader_election.cc:343] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:45.982442 16912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fcfa9f9a5224b69b785589ce292ca3e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:45.948985 17471 consensus_queue.cc:237] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
W20260709 13:05:45.991544 17105 leader_election.cc:343] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:46.002741 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 153241us
I20260709 13:05:46.003993 17570 leader_election.cc:290] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:46.027158 16879 leader_election.cc:304] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:46.037318 16954 leader_election.cc:304] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:46.040210 17600 consensus_queue.cc:1048] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:46.044291 17630 raft_consensus.cc:3060] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.067530 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fcfa9f9a5224b69b785589ce292ca3e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:46.068288 16984 raft_consensus.cc:3060] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.071995 16814 catalog_manager.cc:5697] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:46.073465 17565 leader_election.cc:290] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:46.083357 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:46.084597 17184 tablet_bootstrap.cc:654] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:46.086122 17030 consensus_peers.cc:597] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:46.086654 17030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 76887.5us
I20260709 13:05:46.088639 17185 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:46.089318 17516 raft_consensus.cc:493] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:46.089988 17516 raft_consensus.cc:515] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:46.092379 17516 leader_election.cc:290] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:46.050678 17352 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:46.095930 17352 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:46.098395 17352 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:46.113657 17631 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:46.103889 16811 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58382 (request call id 90) took 1355 ms. Trace:
I20260709 13:05:46.114284 17631 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.099062 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:46.123852 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 86978.1us
I20260709 13:05:46.139655 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:46.140604 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:46.142474 17571 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:46.140869 17064 raft_consensus.cc:2393] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:46.150049 17183 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580: No bootstrap required, opened a new log
I20260709 13:05:46.150732 17183 ts_tablet_manager.cc:1403] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580: Time spent bootstrapping tablet: real 0.320s	user 0.021s	sys 0.006s
I20260709 13:05:46.152383 17631 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:46.153863 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:46.154139 17183 raft_consensus.cc:359] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.154958 17183 raft_consensus.cc:385] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:46.155288 17183 raft_consensus.cc:740] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Initialized, Role: FOLLOWER
I20260709 13:05:46.156042 17183 consensus_queue.cc:260] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:46.156412 17105 leader_election.cc:343] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:46.158672 17183 ts_tablet_manager.cc:1434] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:05:46.163448 17185 tablet_bootstrap.cc:654] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:46.165918 17571 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.168678 17571 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:46.172549 16910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:46.179348 17031 leader_election.cc:343] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:46.181458 17514 raft_consensus.cc:2749] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:46.183226 17028 leader_election.cc:304] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
W20260709 13:05:46.184450 16881 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 81214.8us
I20260709 13:05:46.186282 17428 raft_consensus.cc:2749] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:46.186527 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:46.187247 17428 consensus_queue.cc:1048] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:46.188622 16984 raft_consensus.cc:2468] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:46.197698 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:46.198858 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:46.199719 17131 raft_consensus.cc:2468] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:46.116066 16811 rpcz_store.cc:276] 0709 13:05:44.739456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:44.739683 (+   227us) service_pool.cc:224] Handling call
0709 13:05:46.095100 (+1355417us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:45.388451 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:45.389236 (+   785us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:45.389292 (+    56us) write_op.cc:435] Acquired schema lock
0709 13:05:45.389322 (+    30us) tablet.cc:662] Decoding operations
0709 13:05:45.428735 (+ 39413us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:45.428823 (+    88us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:45.428858 (+    35us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:05:45.429200 (+   342us) tablet.cc:701] Row locks acquired
0709 13:05:45.429231 (+    31us) write_op.cc:260] PREPARE: finished
0709 13:05:45.433727 (+  4496us) write_op.cc:270] Start()
0709 13:05:45.435397 (+  1670us) write_op.cc:276] Timestamp: P: 1783602345429383 usec, L: 0
0709 13:05:45.435436 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 13:05:45.436326 (+   890us) log.cc:844] Serialized 534 byte log entry
0709 13:05:45.842511 (+406185us) op_driver.cc:464] REPLICATION: finished
0709 13:05:45.973824 (+131313us) write_op.cc:301] APPLY: starting
0709 13:05:45.973919 (+    95us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:45.974053 (+   134us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:45.974233 (+   180us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:45.974586 (+   353us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:45.974917 (+   331us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:05:45.974981 (+    64us) write_op.cc:312] APPLY: finished
0709 13:05:45.975299 (+   318us) log.cc:844] Serialized 22 byte log entry
0709 13:05:45.975544 (+   245us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:45.975676 (+   132us) write_op.cc:454] Released schema lock
0709 13:05:45.975763 (+    87us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:45.989467 (+ 13704us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:45.989502 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":69540,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":119937,"apply.run_cpu_time_us":2684,"apply.run_wall_time_us":15575,"num_ops":1,"prepare.queue_time_us":292439,"prepare.run_cpu_time_us":39376,"prepare.run_wall_time_us":180176,"raft.queue_time_us":109030,"raft.run_cpu_time_us":24911,"raft.run_wall_time_us":110411,"replication_time_us":406886,"thread_start_us":419326,"threads_started":4,"wal-append.queue_time_us":153708}]]}
I20260709 13:05:46.209950 17028 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:46.198707 17063 raft_consensus.cc:2468] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 0.
I20260709 13:05:46.216796 17571 raft_consensus.cc:2804] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:46.231194 17630 raft_consensus.cc:2749] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:46.242513 17571 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:46.243563 16988 raft_consensus.cc:2468] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:46.243705 17571 raft_consensus.cc:3060] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:46.239912 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 94541.2us
I20260709 13:05:46.244768 16809 catalog_manager.cc:5697] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:46.253917 17102 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:46.257738 16812 catalog_manager.cc:5697] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } }
W20260709 13:05:46.274410 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 48399.7us
I20260709 13:05:46.296671 16879 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:46.299471 17104 consensus_peers.cc:597] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:46.300213 17104 leader_election.cc:304] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:46.302327 17352 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.313567 17627 raft_consensus.cc:2804] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:46.314164 17627 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:46.314528 17627 raft_consensus.cc:3060] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.330449 17352 raft_consensus.cc:2749] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:46.303951 17570 raft_consensus.cc:2804] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:46.345300 17570 raft_consensus.cc:697] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:46.346318 17570 consensus_queue.cc:237] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:46.303534 17597 consensus_queue.cc:1048] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.069s
I20260709 13:05:46.351655 17185 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:46.352564 17185 ts_tablet_manager.cc:1403] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.264s	user 0.010s	sys 0.040s
I20260709 13:05:46.388490 16989 raft_consensus.cc:1275] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:05:46.393862 17454 consensus_queue.cc:1048] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:46.396867 17104 leader_election.cc:304] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:46.398352 17629 raft_consensus.cc:2804] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:46.393952 16809 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58396 (request call id 74) took 1379 ms. Trace:
I20260709 13:05:46.357008 17185 raft_consensus.cc:359] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:46.402252 17185 raft_consensus.cc:385] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:46.402877 17185 raft_consensus.cc:740] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:46.398913 17629 raft_consensus.cc:697] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:46.407374 17185 consensus_queue.cc:260] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:46.412571 17184 tablet_bootstrap.cc:492] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:46.413199 17184 ts_tablet_manager.cc:1403] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.766s	user 0.013s	sys 0.021s
I20260709 13:05:46.415532 17629 consensus_queue.cc:237] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:46.420212 17184 raft_consensus.cc:359] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:46.421183 17184 raft_consensus.cc:385] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:46.425441 17184 raft_consensus.cc:740] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:46.442615 17454 consensus_queue.cc:1048] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:46.454090 17184 consensus_queue.cc:260] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:46.405918 17571 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.465734 17627 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.468124 17627 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:46.402153 16809 rpcz_store.cc:276] 0709 13:05:45.014637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:45.021575 (+  6938us) service_pool.cc:224] Handling call
0709 13:05:46.393853 (+1372278us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:46.373475 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:46.374077 (+   602us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:46.374117 (+    40us) write_op.cc:435] Acquired schema lock
0709 13:05:46.374140 (+    23us) tablet.cc:662] Decoding operations
0709 13:05:46.374513 (+   373us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:46.374595 (+    82us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:46.374624 (+    29us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:05:46.374812 (+   188us) tablet.cc:701] Row locks acquired
0709 13:05:46.374832 (+    20us) write_op.cc:260] PREPARE: finished
0709 13:05:46.374991 (+   159us) write_op.cc:270] Start()
0709 13:05:46.375104 (+   113us) write_op.cc:276] Timestamp: P: 1783602346374946 usec, L: 0
0709 13:05:46.375127 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:05:46.375764 (+   637us) log.cc:844] Serialized 534 byte log entry
0709 13:05:46.377718 (+  1954us) op_driver.cc:464] REPLICATION: finished
0709 13:05:46.378174 (+   456us) write_op.cc:301] APPLY: starting
0709 13:05:46.378253 (+    79us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:46.378355 (+   102us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:46.378479 (+   124us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:46.378723 (+   244us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:46.378895 (+   172us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:05:46.378983 (+    88us) write_op.cc:312] APPLY: finished
0709 13:05:46.379190 (+   207us) log.cc:844] Serialized 22 byte log entry
0709 13:05:46.379367 (+   177us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:46.379458 (+    91us) write_op.cc:454] Released schema lock
0709 13:05:46.379488 (+    30us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:46.380635 (+  1147us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:46.380659 (+    24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":29126,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":267,"apply.run_cpu_time_us":1630,"apply.run_wall_time_us":1628,"num_ops":1,"prepare.queue_time_us":28616,"prepare.run_cpu_time_us":2641,"prepare.run_wall_time_us":7150,"replication_time_us":2471}]]}
I20260709 13:05:46.469035 17577 raft_consensus.cc:493] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:46.469851 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:46.469774 17577 raft_consensus.cc:515] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.475595 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:46.476336 17135 raft_consensus.cc:2468] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 0.
I20260709 13:05:46.477726 17028 leader_election.cc:304] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:46.481810 17184 ts_tablet_manager.cc:1434] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.068s	user 0.008s	sys 0.000s
I20260709 13:05:46.481964 17572 raft_consensus.cc:2804] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:05:46.482653 17572 raft_consensus.cc:493] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:46.483122 17572 raft_consensus.cc:3060] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.484619 17061 raft_consensus.cc:2393] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:46.491595 17577 leader_election.cc:290] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:46.483130 17184 tablet_bootstrap.cc:492] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b: Bootstrap starting.
I20260709 13:05:46.505679 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:46.507691 17135 raft_consensus.cc:2393] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:46.509737 17185 ts_tablet_manager.cc:1434] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.157s	user 0.010s	sys 0.001s
I20260709 13:05:46.511029 17185 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:46.518563 17571 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:46.502727 17572 raft_consensus.cc:515] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.534003 16910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:46.534611 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:46.535300 17135 raft_consensus.cc:3060] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.535596 16910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:46.547083 16910 raft_consensus.cc:1275] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:05:46.551285 17572 leader_election.cc:290] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:46.557754 17135 raft_consensus.cc:2468] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:46.563261 17597 consensus_queue.cc:1048] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:46.563933 17028 leader_election.cc:304] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:46.565322 17572 raft_consensus.cc:2804] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:05:46.566169 16984 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.566183 17572 raft_consensus.cc:697] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:46.567830 17572 consensus_queue.cc:237] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.561736 17184 tablet_bootstrap.cc:654] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:46.561173 17185 tablet_bootstrap.cc:654] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:46.592418 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:46.652653 16812 catalog_manager.cc:5697] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 2, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } }
I20260709 13:05:46.664597 16812 catalog_manager.cc:5697] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } }
I20260709 13:05:46.662787 17352 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:46.681710 16812 catalog_manager.cc:5697] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:46.690204 17352 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:46.728606 17352 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:46.738906 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:46.743306 17031 leader_election.cc:343] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:46.748836 17031 leader_election.cc:343] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:05:46.746069 17105 leader_election.cc:343] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:46.756922 16910 raft_consensus.cc:3060] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.764325 17454 consensus_queue.cc:1048] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:05:46.758996 17105 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:46.768414 17578 raft_consensus.cc:2749] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:05:46.769235 17185 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:46.769824 17185 ts_tablet_manager.cc:1403] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.259s	user 0.044s	sys 0.014s
I20260709 13:05:46.771225 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:46.772894 17185 raft_consensus.cc:359] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.773733 17185 raft_consensus.cc:385] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:46.773998 17185 raft_consensus.cc:740] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:46.784761 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:46.785759 17185 consensus_queue.cc:260] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:46.791865 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 45980.6us
I20260709 13:05:46.791917 17185 ts_tablet_manager.cc:1434] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 13:05:46.757226 17565 consensus_queue.cc:1048] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20260709 13:05:46.814113 17184 tablet_bootstrap.cc:492] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b: No bootstrap required, opened a new log
I20260709 13:05:46.814723 17184 ts_tablet_manager.cc:1403] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b: Time spent bootstrapping tablet: real 0.332s	user 0.041s	sys 0.033s
I20260709 13:05:46.820979 17184 raft_consensus.cc:359] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:46.821898 17184 raft_consensus.cc:385] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:46.822216 17184 raft_consensus.cc:740] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Initialized, Role: FOLLOWER
I20260709 13:05:46.822909 17184 consensus_queue.cc:260] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:46.826097 16879 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:46.749251 17031 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:46.851311 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 102494us
I20260709 13:05:46.852453 17185 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:46.852231 17571 raft_consensus.cc:2749] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:05:46.853843 16811 catalog_manager.cc:5697] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:46.855886 16811 catalog_manager.cc:5697] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } }
I20260709 13:05:46.860661 16811 catalog_manager.cc:5697] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:46.867959 16811 catalog_manager.cc:5697] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 1 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:46.882149 17184 ts_tablet_manager.cc:1434] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b: Time spent starting tablet: real 0.067s	user 0.008s	sys 0.001s
W20260709 13:05:46.873356 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 13:05:46.875181 17031 leader_election.cc:343] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:05:46.892196 16986 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:46.896678 17352 raft_consensus.cc:2749] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:46.912079 17185 tablet_bootstrap.cc:654] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:46.913115 17597 consensus_queue.cc:1048] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:05:46.950547 17105 consensus_peers.cc:597] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:46.956372 17627 consensus_queue.cc:1048] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 13:05:46.970909 17352 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:46.972918 17352 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:46.972613 16814 catalog_manager.cc:5697] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 0 to 1, leader changed from <none> to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 1 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:46.975128 17352 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:46.993525 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:46.994465 17063 raft_consensus.cc:2393] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
W20260709 13:05:46.998960 17105 consensus_peers.cc:597] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:47.006475 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:47.007411 17138 raft_consensus.cc:2393] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71a10d4adec64d58ba645d68425ed531 in current term 1: Already voted for candidate 40c8156681dd4a7bb2d037afbeb01580 in this term.
I20260709 13:05:47.008833 16879 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:47.014390 17352 raft_consensus.cc:3060] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:47.015026 17185 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:47.015573 17185 ts_tablet_manager.cc:1403] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.173s	user 0.012s	sys 0.020s
I20260709 13:05:47.022655 17185 raft_consensus.cc:359] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:47.023347 17185 raft_consensus.cc:385] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:47.023633 17185 raft_consensus.cc:740] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:47.024278 17185 consensus_queue.cc:260] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:47.032079 17352 raft_consensus.cc:2749] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:47.055076 17185 ts_tablet_manager.cc:1434] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 13:05:47.056306 17185 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:47.065681 17185 tablet_bootstrap.cc:654] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
W20260709 13:05:47.079975 17031 consensus_peers.cc:597] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:47.149732 17185 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:47.150521 17185 ts_tablet_manager.cc:1403] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.094s	user 0.020s	sys 0.019s
I20260709 13:05:47.154048 17185 raft_consensus.cc:359] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:47.154819 17185 raft_consensus.cc:385] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:47.155108 17185 raft_consensus.cc:740] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:47.155786 17185 consensus_queue.cc:260] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:47.159328 17185 ts_tablet_manager.cc:1434] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 13:05:47.160998 17185 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:47.167526 17185 tablet_bootstrap.cc:654] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:47.171496 17428 consensus_queue.cc:1048] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:05:47.198536 17185 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:47.199093 17185 ts_tablet_manager.cc:1403] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.038s	user 0.015s	sys 0.019s
I20260709 13:05:47.202083 17185 raft_consensus.cc:359] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:47.202805 17185 raft_consensus.cc:385] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:47.203083 17185 raft_consensus.cc:740] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:47.203833 17185 consensus_queue.cc:260] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:47.206324 17185 ts_tablet_manager.cc:1434] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:05:47.207756 17185 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:47.214058 17185 tablet_bootstrap.cc:654] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:47.255429 17185 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:47.255980 17185 ts_tablet_manager.cc:1403] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.048s	user 0.014s	sys 0.016s
I20260709 13:05:47.260625 17185 raft_consensus.cc:359] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:47.261351 17185 raft_consensus.cc:385] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:47.261677 17185 raft_consensus.cc:740] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:47.262504 17185 consensus_queue.cc:260] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:47.264904 17185 ts_tablet_manager.cc:1434] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:05:47.266656 17185 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:47.280764 17185 tablet_bootstrap.cc:654] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:47.299978 17185 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:47.300625 17185 ts_tablet_manager.cc:1403] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.034s	user 0.009s	sys 0.015s
I20260709 13:05:47.304072 17185 raft_consensus.cc:359] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:47.304924 17185 raft_consensus.cc:385] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:47.305287 17185 raft_consensus.cc:740] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:47.306190 17185 consensus_queue.cc:260] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:47.308673 17185 ts_tablet_manager.cc:1434] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:05:47.310516 17185 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:47.312320 17638 raft_consensus.cc:493] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:47.312904 17638 raft_consensus.cc:515] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:47.322681 17185 tablet_bootstrap.cc:654] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Neither blocks nor log segments found. Creating new log.
I20260709 13:05:47.325740 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:47.333854 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:47.342547 16915 raft_consensus.cc:3060] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:47.343564 16881 leader_election.cc:304] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:47.340548 17638 leader_election.cc:290] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:47.345518 17352 raft_consensus.cc:2749] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:47.362700 17185 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: No bootstrap required, opened a new log
I20260709 13:05:47.363307 17185 ts_tablet_manager.cc:1403] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.053s	user 0.011s	sys 0.020s
I20260709 13:05:47.366744 17185 raft_consensus.cc:359] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:47.367520 17185 raft_consensus.cc:385] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:05:47.367823 17185 raft_consensus.cc:740] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:47.368705 17185 consensus_queue.cc:260] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:47.371735 17185 ts_tablet_manager.cc:1434] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:05:47.477169 16915 raft_consensus.cc:3060] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:47.480206 17638 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:47.480996 17638 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:47.484728 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:47.486624 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:47.487020 16913 raft_consensus.cc:3060] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:47.488456 16879 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:47.489343 17640 raft_consensus.cc:2749] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:05:47.492933 17638 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:47.494493 16910 raft_consensus.cc:3060] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:47.601822 16910 raft_consensus.cc:3060] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:05:47.713490 16910 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:05:47.930198 16862 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:05:48.546893 17454 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:48.548032 17454 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:48.552595 17454 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:48.553174 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:48.553514 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:48.553905 16913 raft_consensus.cc:2468] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:48.554280 17063 raft_consensus.cc:2468] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:48.555182 17105 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:05:48.555904 17454 raft_consensus.cc:2804] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:48.556255 17454 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:48.556532 17454 raft_consensus.cc:3060] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:48.561522 17454 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:48.563191 17454 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:48.564090 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:48.564276 16913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:48.564693 17063 raft_consensus.cc:3060] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:48.564936 16913 raft_consensus.cc:3060] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:48.571691 17063 raft_consensus.cc:2468] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:48.574189 17102 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:48.575848 16913 raft_consensus.cc:2468] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:48.575973 17570 raft_consensus.cc:2804] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:48.578055 17570 raft_consensus.cc:697] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:48.578970 17570 consensus_queue.cc:237] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:48.593257 16812 catalog_manager.cc:5697] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 0 to 2, leader changed from <none> to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:49.492659 17063 raft_consensus.cc:1275] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:05:49.843660 17699 consensus_queue.cc:1048] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260709 13:05:49.870719 17694 raft_consensus.cc:493] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:49.871596 17694 raft_consensus.cc:515] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:49.894093 17692 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:49.894804 17692 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:49.897891 17692 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:49.920044 17698 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:49.920646 17698 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:49.922817 17698 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:49.929961 16915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
W20260709 13:05:49.938722 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 51301.3us
I20260709 13:05:49.943138 16910 raft_consensus.cc:1275] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:05:49.946485  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
I20260709 13:05:49.949502 17708 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:49.950556 17708 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:50.112962 17709 raft_consensus.cc:493] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.115056 17709 raft_consensus.cc:515] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:50.099709 17690 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.117569 17690 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.118522 17707 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.119403 17707 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.113260 17702 raft_consensus.cc:493] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.186219 17714 raft_consensus.cc:493] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.187064 17714 raft_consensus.cc:515] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.189819 17711 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.190740 17711 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.193867 17711 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:50.196004 17715 raft_consensus.cc:493] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.198163 17715 raft_consensus.cc:515] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.178932 17700 raft_consensus.cc:493] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.205826 17700 raft_consensus.cc:515] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:50.207268 17715 leader_election.cc:290] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:50.204373 17702 raft_consensus.cc:515] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.113325 17698 raft_consensus.cc:493] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.214457 17698 raft_consensus.cc:515] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.179593 17704 raft_consensus.cc:493] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:50.220815 17704 raft_consensus.cc:515] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:50.248685 17701 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.254571 17701 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.249456 17715 raft_consensus.cc:493] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
W20260709 13:05:50.350529 16957 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.000s	sys 0.022s
W20260709 13:05:50.354456 16957 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.000s	sys 0.022s
I20260709 13:05:50.258698 17704 leader_election.cc:290] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:50.269111 17711 raft_consensus.cc:493] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.264638 17701 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:50.341205 17703 raft_consensus.cc:493] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.611660 17722 raft_consensus.cc:493] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.615124 17454 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:50.397293 17699 raft_consensus.cc:493] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.263597 17712 raft_consensus.cc:493] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.616437 17693 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:50.671077 17454 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:50.263851 17715 raft_consensus.cc:515] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.614658 17706 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.675734 17693 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:50.798799 17720 raft_consensus.cc:493] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:50.799038 17715 leader_election.cc:290] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:50.799682 17693 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:50.676291 17703 raft_consensus.cc:515] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.673645 17708 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:50.799459 17720 raft_consensus.cc:515] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:50.803982 17703 leader_election.cc:290] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:50.620188 17711 raft_consensus.cc:515] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.893272 17570 raft_consensus.cc:493] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:50.794811 17716 raft_consensus.cc:493] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.902529 17454 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:50.893488 17719 raft_consensus.cc:493] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.672676 17722 raft_consensus.cc:515] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.673171 17699 raft_consensus.cc:515] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.894155 17570 raft_consensus.cc:515] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.945576 17719 raft_consensus.cc:515] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:50.891929 17703 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.788633 17712 raft_consensus.cc:515] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.973115 17723 raft_consensus.cc:493] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:50.956667 17703 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:50.973949 17723 raft_consensus.cc:515] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.976478 17723 leader_election.cc:290] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:50.963951 17704 raft_consensus.cc:493] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.979794 17704 raft_consensus.cc:515] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.982312 17704 leader_election.cc:290] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:50.964191 17715 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.983525 17715 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.984808 16913 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36872 (request call id 199) took 1220 ms. Trace:
I20260709 13:05:50.976650 17703 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:50.986222 17703 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.987023 17703 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.988931 17720 leader_election.cc:290] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:50.989919 17703 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:50.810654 17706 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:50.955503 17721 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:50.997776 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:50.998376 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:50.963536 17701 raft_consensus.cc:493] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:50.994788 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:50.985255 16913 rpcz_store.cc:276] 0709 13:05:49.764035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:49.764530 (+   495us) service_pool.cc:224] Handling call
0709 13:05:50.805071 (+1040541us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b5800128798. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:05:50.805330 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:05:50.921342 (+116012us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:05:50.921392 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:05:50.921478 (+    86us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:05:50.921527 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:05:50.984519 (+ 62992us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400382554. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:05:50.984580 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:05:50.984710 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":270871296}
I20260709 13:05:50.964246 17725 raft_consensus.cc:493] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.991662 17728 raft_consensus.cc:493] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:50.931993 17716 raft_consensus.cc:515] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.003911 17721 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:50.973167 17570 leader_election.cc:290] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.007700 17711 leader_election.cc:290] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.008584 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.010497 17722 leader_election.cc:290] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.087512 17694 leader_election.cc:290] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.088145 17694 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.088723 17694 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.006630 17713 raft_consensus.cc:493] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.029385 17712 leader_election.cc:290] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.012682 17725 raft_consensus.cc:515] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.040652 17731 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.092736 17731 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.093590 16984 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33864 (request call id 214) took 1164 ms. Trace:
I20260709 13:05:51.093848 17725 leader_election.cc:290] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.011906 17706 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.051690 17721 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.028728 17719 leader_election.cc:290] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.095817 17731 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.094151 16984 rpcz_store.cc:276] 0709 13:05:49.928868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:49.931922 (+  3054us) service_pool.cc:224] Handling call
0709 13:05:51.092643 (+1160721us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b5800004298. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:05:51.092904 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:05:51.093122 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:05:51.093170 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:05:51.093229 (+    59us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:05:51.093272 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:05:51.093344 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:05:51.093495 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2745216}
I20260709 13:05:51.013010 17708 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.110683 17708 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.102417 17694 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.107679 17734 raft_consensus.cc:493] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.027732 17699 leader_election.cc:290] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.023985 17728 raft_consensus.cc:515] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.114153 17708 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.115425 17728 leader_election.cc:290] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.061951 17709 leader_election.cc:290] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.012329 17715 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.026993 17716 leader_election.cc:290] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.053241 17714 leader_election.cc:290] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.098311 17713 raft_consensus.cc:515] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.119299 17713 leader_election.cc:290] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.039940 17703 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.120272 17703 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.122962 17703 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.073511 17710 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.123773 17710 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.126430 17710 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.015731 17701 raft_consensus.cc:515] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.128882 17701 leader_election.cc:290] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.128986 17702 leader_election.cc:290] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.074553 17732 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:51.130330 17732 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.130676 17734 raft_consensus.cc:515] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.133030 17732 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.038627 17720 raft_consensus.cc:493] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.133842 17734 leader_election.cc:290] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.133949 17720 raft_consensus.cc:515] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.137822 17720 leader_election.cc:290] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.138450 17690 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.140445 17732 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.142886 17707 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.144245 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.141040 17732 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.144920 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.145859 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.146479 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.147171 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.145856 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.149032 17060 raft_consensus.cc:2468] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:51.150238 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.150301 16879 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
W20260709 13:05:51.151600 17104 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.028s	sys 0.044s
W20260709 13:05:51.151942 17104 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.028s	sys 0.044s
I20260709 13:05:51.151860 17720 raft_consensus.cc:2804] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.152534 17720 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.149222 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.153539 17056 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:51.154331 17692 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:51.154953 17706 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.154837 17692 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.155737 17063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60996 (request call id 227) took 1396 ms. Trace:
I20260709 13:05:51.155515 17706 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.156170 17063 rpcz_store.cc:276] 0709 13:05:49.759127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:49.759366 (+   239us) service_pool.cc:224] Handling call
0709 13:05:51.154908 (+1395542us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b5800677a98. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f14069b1db1 00007f1406af1b71 00007f1406af1a22 00007f140fa0ac09 00007f140eb64ee0 00007f140ec2e0ab 00007f140ec2e131 00007f140ebeffc0 00007f140ebda929 00007f140ebda3d2 00007f1408151bd4
0709 13:05:51.155125 (+   217us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:05:51.155359 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:05:51.155388 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:05:51.155431 (+    43us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:05:51.155463 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:05:51.155521 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:05:51.155654 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10602752}
I20260709 13:05:51.157799 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.158517 16987 raft_consensus.cc:2468] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:51.146651 17732 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.160426 17696 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.160941 17696 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.156922 17692 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.146605 16988 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:51.163669 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.164357 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.164971 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.165045 17059 raft_consensus.cc:2468] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:51.166723 16879 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.153170 17720 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.165745 17056 raft_consensus.cc:2468] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:51.165578 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.168385 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.168870 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.169546 17707 raft_consensus.cc:2804] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.170050 17707 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.170593 17707 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.170122 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.166195 17706 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.173466 16954 leader_election.cc:304] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.174211 17701 raft_consensus.cc:2804] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.174685 17701 raft_consensus.cc:493] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.175164 17701 raft_consensus.cc:3060] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.175639 17030 leader_election.cc:304] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.182854 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.186436 17696 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.188040 17714 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.188604 17714 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.188958 17570 raft_consensus.cc:493] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.206781 17736 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:51.208835 17707 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.209483 17699 raft_consensus.cc:493] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.223168 17699 raft_consensus.cc:515] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.224607 17707 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.210575 17727 raft_consensus.cc:493] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.225971 17736 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.226141 17727 raft_consensus.cc:515] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.210059 17697 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.228955 17736 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.229306 17697 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.232973 17697 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.233595 17699 leader_election.cc:290] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.212031 16912 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36856 (request call id 284) took 1743 ms. Trace:
I20260709 13:05:51.169615 17056 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:51.213282 17724 raft_consensus.cc:493] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.215215 17692 consensus_queue.cc:1048] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:05:51.235432 17103 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.006s	sys 0.004s
W20260709 13:05:51.285993 17103 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.006s	sys 0.005s
I20260709 13:05:51.234211 17727 leader_election.cc:290] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.240167 17698 leader_election.cc:290] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
W20260709 13:05:51.216235 17105 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20260709 13:05:51.217119 17690 raft_consensus.cc:493] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.245008 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.288408 17690 raft_consensus.cc:515] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.219417 17714 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.291384 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.291680 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.292698 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.292954 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.245738 17734 raft_consensus.cc:2804] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.293339 16988 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:51.294322 17734 raft_consensus.cc:493] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.294864 17734 raft_consensus.cc:3060] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.295073 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.296130 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.296854 16881 leader_election.cc:304] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.296535 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.255493 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.298978 16881 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:51.298071 17454 raft_consensus.cc:493] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:51.299926 17707 raft_consensus.cc:2749] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.259366 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.300230 17454 raft_consensus.cc:515] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.262733 17700 leader_election.cc:290] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.264089 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.302752 17454 leader_election.cc:290] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.265705 17722 raft_consensus.cc:493] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.304036 17722 raft_consensus.cc:515] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.252413 17728 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.253014 17570 raft_consensus.cc:515] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.306241 17722 leader_election.cc:290] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.267033 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.308035 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.308318 17734 raft_consensus.cc:515] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.267457 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e77860bb944a5e97539267e7d0f9ba" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.269543 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.310318 17701 raft_consensus.cc:515] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.310685 17734 leader_election.cc:290] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.306025 17728 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.279116 16954 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.252784 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:51.325203 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 46214.8us
I20260709 13:05:51.307974 17570 leader_election.cc:290] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.242798 17709 raft_consensus.cc:493] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.315096 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.326208 17709 raft_consensus.cc:515] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.326679 17712 raft_consensus.cc:2804] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:05:51.245570 17030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 70012.1us
I20260709 13:05:51.327126 17712 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.327545 17712 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.327726 17030 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.328514 17709 leader_election.cc:290] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.328881 17714 raft_consensus.cc:2804] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.329331 17714 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.329782 17714 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.331965 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.268514 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.252784 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2521b7f56acb452087e4eccf58dd0f4a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.252295 17731 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.314680 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.245994 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.311837 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.279383 17697 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.314226 17705 raft_consensus.cc:2804] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:51.312109 17064 raft_consensus.cc:1240] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
W20260709 13:05:51.270838 17028 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.187s	user 0.035s	sys 0.038s
I20260709 13:05:51.326452 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.333360 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:51.344844 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 122751us
I20260709 13:05:51.339205 16879 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
W20260709 13:05:51.346650 17028 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.263s	user 0.036s	sys 0.038s
I20260709 13:05:51.278640 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e77860bb944a5e97539267e7d0f9ba" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.257216 16912 rpcz_store.cc:276] 0709 13:05:49.468273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:50.198941 (+730668us) service_pool.cc:224] Handling call
0709 13:05:50.199303 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:05:50.249729 (+ 50426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:05:50.249762 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:05:50.249804 (+    42us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:05:50.249829 (+    25us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:05:51.211623 (+961794us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x7b5400383454. stack: 00007f1406af7343 00007f1406af7240 00007f14064861d4 00007f14101c4304 00007f14101c4299 00007f1409b7f571 00007f1409c16e01 00007f1409c1583e 00007f140ebda419 00007f1408151bd4 00007f1408151b62 00007f1408151a91 00007f1408151a0c 00007f1408150cc2 00007f1407f5148c 00007f1407f508c6
0709 13:05:51.211686 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:05:51.211964 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115463552}
I20260709 13:05:51.348939 17454 raft_consensus.cc:493] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.353969 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.354318 17454 raft_consensus.cc:515] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.356804 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.282956 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.359771 17697 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.360776 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.361613 16986 raft_consensus.cc:2393] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 2: Already voted for candidate c9521ca6efa94c3e9f074df69170cd5b in this term.
I20260709 13:05:51.362565 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.362634 17697 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.353076 17706 raft_consensus.cc:493] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.364792 17706 raft_consensus.cc:515] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.314025 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.309247 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.362993 17454 leader_election.cc:290] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.259671 17724 raft_consensus.cc:515] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.245589 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.374155 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.339207 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.386943 17133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.388048 17133 raft_consensus.cc:2468] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:51.389545 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.390420 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.391628 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.377724 17454 raft_consensus.cc:493] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.335202 17728 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.375938 17713 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.366837 17705 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.393606 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.394076 17705 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:51.394001 17713 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.394747 17137 raft_consensus.cc:2468] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:51.379034 17700 raft_consensus.cc:493] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.372773 17709 raft_consensus.cc:2804] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.277858 17058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.396727 17700 raft_consensus.cc:515] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.399191 17700 leader_election.cc:290] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.399925 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.381062 17703 raft_consensus.cc:493] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.368422 17707 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.385784 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.270638 16982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.386348 17701 leader_election.cc:290] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.379938 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.371673 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.401849 16989 raft_consensus.cc:1240] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:51.336585 16991 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.393687 17454 raft_consensus.cc:515] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.377092 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:51.410071 16879 leader_election.cc:304] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.337193 17731 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.409966 17699 raft_consensus.cc:493] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:51.277275 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.416649 17064 raft_consensus.cc:2468] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:51.423203 17028 leader_election.cc:304] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.440340 17028 leader_election.cc:304] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.396719 17709 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.411552 17694 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.412436 17703 raft_consensus.cc:515] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.451512 17733 raft_consensus.cc:493] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.452143 17733 raft_consensus.cc:515] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.453006 17703 leader_election.cc:290] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.426038 17719 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.455786 16879 leader_election.cc:304] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.456089 17719 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.456558 17714 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.457304 17705 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.458583 17719 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.458782 17714 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.454547 17735 raft_consensus.cc:2749] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.459235 17719 raft_consensus.cc:2749] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.437938 17454 leader_election.cc:290] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.457149 16879 leader_election.cc:304] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.460765 17705 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.450807 17727 raft_consensus.cc:493] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.406113 17700 raft_consensus.cc:493] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.461427 17705 raft_consensus.cc:2749] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.461546 17727 raft_consensus.cc:515] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.461712 17700 raft_consensus.cc:515] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.453891 17737 raft_consensus.cc:2804] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.446972 17702 raft_consensus.cc:493] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.464545 17737 raft_consensus.cc:493] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.465327 17737 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.453799 17694 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.466286 17713 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.468120 17694 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.439174 17725 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.468842 17725 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.459905 17690 leader_election.cc:290] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.430944 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
W20260709 13:05:51.470489 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 254253us
I20260709 13:05:51.443574 17028 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.454758 17724 leader_election.cc:290] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.470906 16879 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.483525 16879 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.415361 17132 raft_consensus.cc:2468] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:51.464725 17702 raft_consensus.cc:515] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.433004 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.412267 17707 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.448741 17709 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.448570 17734 raft_consensus.cc:2749] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.449252 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.466145 17700 leader_election.cc:290] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
W20260709 13:05:51.497808 17031 consensus_peers.cc:597] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.461716 17731 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.458632 17733 leader_election.cc:290] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.456004 17720 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.493929 17690 raft_consensus.cc:2804] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.489147 17701 raft_consensus.cc:493] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.497020 17735 raft_consensus.cc:2804] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.494529 16982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.493355 17705 raft_consensus.cc:2749] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.504940 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.508002 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fcfa9f9a5224b69b785589ce292ca3e" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.560465 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.561728 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f4a7e79870e405e9938b891f964fd1d" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.562825 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.563879 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.564917 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.533981 17028 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
W20260709 13:05:51.532528 16957 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 112669us
I20260709 13:05:51.474463 17699 raft_consensus.cc:515] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.505389 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.526794 17707 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.524534 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:51.522130 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.550030 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.550288 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.555913 17702 leader_election.cc:290] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.557947 16982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.534044 17057 raft_consensus.cc:2393] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b in current term 2: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:51.541064 17701 raft_consensus.cc:515] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.502262 17706 leader_election.cc:290] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:51.571825 17705 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.571902 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.507825 17727 leader_election.cc:290] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.574791 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:51.513639 17139 raft_consensus.cc:2468] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:51.577654 17720 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.523495 17725 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.534461 17690 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.537992 17735 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.531849 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.574884 17719 raft_consensus.cc:493] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
W20260709 13:05:51.601315 17031 consensus_peers.cc:597] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.607393 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 97939us
I20260709 13:05:51.575601 17694 raft_consensus.cc:2804] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.584976 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.605173 16990 raft_consensus.cc:2468] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:51.619331 17731 raft_consensus.cc:493] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.619771 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.620038 17731 raft_consensus.cc:515] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.620560 16990 raft_consensus.cc:2468] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:51.622717 17699 leader_election.cc:290] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.606246 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.602181 17733 raft_consensus.cc:493] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.624505 17733 raft_consensus.cc:515] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:51.584796 17060 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
W20260709 13:05:51.616660 17031 consensus_peers.cc:597] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.609931 17735 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.632340 16954 leader_election.cc:304] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:51.659580 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 47550.4us
I20260709 13:05:51.649662 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.604739 17690 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.643232 16881 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:51.618378 17706 consensus_queue.cc:1059] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 13:05:51.615480 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.625300 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.663954 17692 raft_consensus.cc:3055] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:51.664347 17692 raft_consensus.cc:740] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:51.664933 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.665452 17692 consensus_queue.cc:260] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.666177 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.666782 17692 raft_consensus.cc:3060] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.667061 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.667394 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:51.668226 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3fbc4f00185436cbce1d8e24e6d1e3a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.640829 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.618532 17712 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.637143 17104 leader_election.cc:304] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:51.664250 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.671190 17031 leader_election.cc:336] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.674446 16954 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.675830 17707 raft_consensus.cc:493] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.676373 17707 raft_consensus.cc:515] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.678692 17722 raft_consensus.cc:2749] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:51.613166 17702 raft_consensus.cc:493] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.679520 17702 raft_consensus.cc:515] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.680575 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.670469 17715 raft_consensus.cc:493] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:51.681788 17702 leader_election.cc:290] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.669946 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.626453 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b240339eb134181914367d7072b86b6" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.682308 17720 raft_consensus.cc:2804] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.682842 17720 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.683135 16954 leader_election.cc:304] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.683261 17720 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.684170 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fcfa9f9a5224b69b785589ce292ca3e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.685853 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.689535 17696 raft_consensus.cc:2804] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.689977 17696 raft_consensus.cc:493] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.690244 17696 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.690689 17721 raft_consensus.cc:2749] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.626605 17733 leader_election.cc:290] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.616461 17719 raft_consensus.cc:515] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.681501 17715 raft_consensus.cc:515] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.696828 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d582455c79c94516b94ee97bd42c5078" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.698084 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.674444 17031 leader_election.cc:304] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.700515 16879 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.701941 17028 leader_election.cc:304] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.678664 17707 leader_election.cc:290] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.615108 17703 raft_consensus.cc:493] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.599444 17705 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.629640 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:51.614084 17694 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.708055 17725 raft_consensus.cc:2804] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.701650 17722 raft_consensus.cc:2749] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.707538 16881 leader_election.cc:304] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.710482 17028 leader_election.cc:304] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.711052 17710 raft_consensus.cc:2749] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.724736 17725 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.713927 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.704949 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.715067 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.712880 17133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.716276 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.716356 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.719033 17733 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.636299 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.622720 17731 leader_election.cc:290] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.690968 17720 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:51.664289 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.711229 16982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.729161 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2784691feee4fdc945ee8ea311ea7da" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.729499 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 67672.7us
W20260709 13:05:51.728531 17105 consensus_peers.cc:597] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.729650 17028 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.730196 17697 raft_consensus.cc:2749] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.726583 16985 raft_consensus.cc:1240] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:51.713711 17696 raft_consensus.cc:515] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.739547 17732 raft_consensus.cc:493] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.743247 17733 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.745697 16881 leader_election.cc:304] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.746006 17705 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.731591 17736 raft_consensus.cc:2749] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.728694 17694 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.731861 17725 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:51.756188 17031 leader_election.cc:336] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.769480 17031 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.758435 17028 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.731653 17702 raft_consensus.cc:2749] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:51.757844 17105 consensus_peers.cc:597] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.764809 17709 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:51.771272 17105 consensus_peers.cc:597] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.771921 17105 leader_election.cc:336] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.772311 17105 leader_election.cc:304] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.763312 17732 raft_consensus.cc:515] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.773278 17706 raft_consensus.cc:2749] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.773646 17719 leader_election.cc:290] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
W20260709 13:05:51.774718 17105 leader_election.cc:336] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.775043 17732 leader_election.cc:290] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:51.775549 17105 leader_election.cc:336] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.773209 17137 raft_consensus.cc:1240] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Rejecting Update request from peer f83f5ad22c2547b29c99ae897ea9b61b for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:51.775889 17064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
W20260709 13:05:51.776116 17105 consensus_peers.cc:597] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.776721 17105 leader_election.cc:336] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.772729 17709 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.776925 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.777789 17728 raft_consensus.cc:2749] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.778177 17712 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.778708 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.766181 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.779460 17737 raft_consensus.cc:515] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.780260 17057 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:51.782598 17720 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.782783 16982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:51.784653 16957 consensus_peers.cc:597] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.784991 16879 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.773530 17696 leader_election.cc:290] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.768662 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.777292 17105 leader_election.cc:336] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.794071 17065 raft_consensus.cc:1240] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer c9521ca6efa94c3e9f074df69170cd5b for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:51.803642 17105 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.804450 17727 raft_consensus.cc:2749] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:51.804829 17105 consensus_peers.cc:597] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.787679 17701 leader_election.cc:290] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.810715 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e32f5ae45c549ef88703600d6a9cde3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.802029 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.802320 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.787979 17737 leader_election.cc:290] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.788946 17715 leader_election.cc:290] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:51.765110 17714 raft_consensus.cc:2804] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.725378 17703 raft_consensus.cc:515] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
W20260709 13:05:51.805672 17031 leader_election.cc:336] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.822104 17031 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.824460 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.824831 17715 consensus_queue.cc:1059] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:51.825515 17133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:51.815977 17728 raft_consensus.cc:493] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:51.826351 17715 raft_consensus.cc:3055] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:51.786994 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:51.826967 17715 raft_consensus.cc:740] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:51.828613 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:51.828788 17715 consensus_queue.cc:260] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.830874 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.826856 17728 raft_consensus.cc:515] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.788786 17720 raft_consensus.cc:2804] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.834695 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:51.790242 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.768437 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91680aaf7f642fb8b9697eaa08d67ac" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.798470 17734 raft_consensus.cc:2749] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.833884 17735 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.834818 17720 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.836185 17728 leader_election.cc:290] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.816136 17714 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:51.836557 17720 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.836838 17714 raft_consensus.cc:3060] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.806845 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.839600 16881 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 70862us
W20260709 13:05:51.840435 17031 leader_election.cc:336] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:05:51.785362 16957 consensus_peers.cc:597] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.805524 17105 consensus_peers.cc:597] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.840914 17031 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.807204 17732 raft_consensus.cc:2749] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:05:51.841305 16957 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 56031.3us
I20260709 13:05:51.840793 16881 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:51.845070 16957 leader_election.cc:336] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.846225 16957 leader_election.cc:304] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:51.844727 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f90b95672134fac819741b525d2931b" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.848832 17105 consensus_peers.cc:597] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.826761 17133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.849885 17105 consensus_peers.cc:597] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.830942 17715 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.849653 17701 raft_consensus.cc:2749] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.861639 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:51.862398 17105 consensus_peers.cc:597] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.862936 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.863765 17105 leader_election.cc:336] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.852802 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b78448a359344202ad036ddd40ccee93" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:51.853127 17031 consensus_peers.cc:597] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.854245 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fcfa9f9a5224b69b785589ce292ca3e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.854396 17697 raft_consensus.cc:2749] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.856459 17733 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.870599 17703 leader_election.cc:290] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:51.858067 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.871001 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.871891 17139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.872207 17137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd439fd5f4d4479f9d2c2003abed1c4c" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.860808 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.843024 17734 raft_consensus.cc:2749] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.876332 17136 raft_consensus.cc:2468] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 2.
I20260709 13:05:51.879199 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.847997 16957 consensus_peers.cc:597] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.880196 16957 leader_election.cc:336] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.880614 16957 leader_election.cc:304] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.881796 16954 leader_election.cc:304] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.793687 17690 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.842185 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.866309 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.857764 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.882108 16881 leader_election.cc:304] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.893553 16881 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.895376 17028 leader_election.cc:304] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.883049 17105 leader_election.cc:304] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
W20260709 13:05:51.883050 16957 consensus_peers.cc:597] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.883181 17102 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.864696 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:51.898316 16957 consensus_peers.cc:597] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.852059 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5613a2ba864f379a89f716babe1879" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.885730 17701 raft_consensus.cc:2749] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.885916 17721 raft_consensus.cc:2804] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:51.900091 17721 raft_consensus.cc:697] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
W20260709 13:05:51.900633 16957 leader_election.cc:336] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.886442 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3fbc4f00185436cbce1d8e24e6d1e3a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:51.901109 16957 leader_election.cc:304] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:51.844892 17728 raft_consensus.cc:2749] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.901244 17721 consensus_queue.cc:237] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:51.902376 16957 consensus_peers.cc:597] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.884744 16954 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:51.903033 16957 consensus_peers.cc:597] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.903571 16957 consensus_peers.cc:597] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.904086 16957 consensus_peers.cc:597] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:51.904361 17722 raft_consensus.cc:2749] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.905154 17722 raft_consensus.cc:2749] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:05:51.905367 16957 leader_election.cc:336] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.906212 16957 leader_election.cc:304] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:51.906883 17722 raft_consensus.cc:2749] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.907980 17692 raft_consensus.cc:2749] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.888605 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:51.897353 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.897487 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:51.909049 17061 raft_consensus.cc:2468] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 1.
I20260709 13:05:51.910159 17734 raft_consensus.cc:2749] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.913267 17102 leader_election.cc:304] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.914985 17102 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.918711 17062 raft_consensus.cc:1240] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer c9521ca6efa94c3e9f074df69170cd5b for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:51.920365 17696 raft_consensus.cc:2749] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:05:51.908541 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 44811.4us
W20260709 13:05:51.923234 17105 leader_election.cc:336] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.923784 17105 leader_election.cc:304] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
W20260709 13:05:51.932886 17104 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:05:51.925729 17734 consensus_queue.cc:1059] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:05:51.926187 17692 raft_consensus.cc:2749] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.956365 17728 raft_consensus.cc:2749] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:05:51.956151 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 61759.1us
W20260709 13:05:51.976516 16957 leader_election.cc:336] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.977046 16957 leader_election.cc:304] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:51.977298 17736 raft_consensus.cc:3055] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:51.977607 16879 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:51.977841 17736 raft_consensus.cc:740] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:51.978164 17721 raft_consensus.cc:2749] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.978708 17733 raft_consensus.cc:2804] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:51.979175 17733 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:51.978827 17736 consensus_queue.cc:260] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.956151 17696 raft_consensus.cc:2749] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.893065 17703 raft_consensus.cc:2749] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.957763 17706 raft_consensus.cc:2749] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.868346 17694 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.962472 17735 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.963792 16881 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:51.964262 17102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 49393.3us
W20260709 13:05:51.964807 17105 consensus_peers.cc:597] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:51.987710 17105 leader_election.cc:336] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.987879 17102 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:51.958657 17698 consensus_queue.cc:1059] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 13:05:51.967638 17690 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.972216 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:51.972937 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:51.970367 17744 consensus_queue.cc:1059] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:05:51.979579 17733 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:51.927435 17720 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:51.994252 17701 raft_consensus.cc:3055] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:51.994699 17701 raft_consensus.cc:740] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:51.994823 17102 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:51.980166 16957 leader_election.cc:336] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.995702 17727 raft_consensus.cc:2749] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:51.986537 17030 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:05:51.995739 17701 consensus_queue.cc:260] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:51.988149 17105 leader_election.cc:304] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.991318 17719 raft_consensus.cc:2749] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.997709 17727 raft_consensus.cc:2749] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.995679 17720 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:51.989110 16881 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 62671.3us
I20260709 13:05:51.993232 17692 raft_consensus.cc:2749] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:51.998039 17105 leader_election.cc:336] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.998032 17059 raft_consensus.cc:1240] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer c9521ca6efa94c3e9f074df69170cd5b for earlier term 1. Current term is 2. Ops: []
W20260709 13:05:51.966296 17031 leader_election.cc:336] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.998973 17105 leader_election.cc:304] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:51.999174 17031 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:52.000301 17105 consensus_peers.cc:597] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.000607 17454 raft_consensus.cc:2749] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.000502 17031 consensus_peers.cc:597] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.001035 17720 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:51.996338 16957 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:52.000882 17105 leader_election.cc:336] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:05:52.003062 17105 consensus_peers.cc:597] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.003826 17028 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
W20260709 13:05:52.001816 17031 leader_election.cc:336] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.004609 17031 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:52.000852 17734 raft_consensus.cc:2749] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.003511 17105 consensus_peers.cc:597] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:52.006278 17105 leader_election.cc:336] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:05:52.007023 17105 leader_election.cc:336] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:05:52.007272 16957 leader_election.cc:336] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.007437 17105 leader_election.cc:304] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.007761 16957 leader_election.cc:304] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:52.007588 17714 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:52.008667 17058 raft_consensus.cc:1240] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer c9521ca6efa94c3e9f074df69170cd5b for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:52.009092 17721 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:52.009816 17721 raft_consensus.cc:2749] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.010955 16957 leader_election.cc:336] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.011355 16957 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:52.005630 17720 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.014212 17720 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:51.987818 17736 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.015636 17721 raft_consensus.cc:2749] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.018251 16957 leader_election.cc:336] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.018690 16957 leader_election.cc:304] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:52.019404 17721 raft_consensus.cc:2749] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.008637 17692 raft_consensus.cc:2749] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.020789 16957 leader_election.cc:336] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.021234 16957 leader_election.cc:304] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:52.021478 17012 debug-util.cc:398] Leaking SignalData structure 0x7b08002873a0 after lost signal to thread 7484
I20260709 13:05:52.022006 17721 raft_consensus.cc:2749] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.022866 17012 debug-util.cc:398] Leaking SignalData structure 0x7b08001c61a0 after lost signal to thread 16848
W20260709 13:05:52.023255 16957 consensus_peers.cc:597] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:52.024273 17012 debug-util.cc:398] Leaking SignalData structure 0x7b0800207c00 after lost signal to thread 16940
W20260709 13:05:52.024905 16957 leader_election.cc:336] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:51.997345 17701 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.024863 17733 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.025552 16957 leader_election.cc:304] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.027402 17721 raft_consensus.cc:2749] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:51.985854 17725 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.026650 17694 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:52.028523 16957 leader_election.cc:336] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.048856 16957 leader_election.cc:304] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:05:52.025749 17012 debug-util.cc:398] Leaking SignalData structure 0x7b080024c1a0 after lost signal to thread 17015
W20260709 13:05:52.052531 17012 debug-util.cc:398] Leaking SignalData structure 0x7b08003f5340 after lost signal to thread 17089
I20260709 13:05:52.052575 17712 consensus_queue.cc:1059] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:05:52.054700 17712 raft_consensus.cc:3055] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:52.055075 17712 raft_consensus.cc:740] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:52.055785 17711 raft_consensus.cc:2749] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.055888 17712 consensus_queue.cc:260] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:52.058337 17012 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd7c0 after lost signal to thread 17163
W20260709 13:05:52.058802 17012 debug-util.cc:398] Leaking SignalData structure 0x7b0800310980 after lost signal to thread 17428
W20260709 13:05:52.059080 17012 debug-util.cc:398] Leaking SignalData structure 0x7b0800143660 after lost signal to thread 17565
W20260709 13:05:52.059299 17012 debug-util.cc:398] Leaking SignalData structure 0x7b0800328000 after lost signal to thread 17571
I20260709 13:05:51.997269 17102 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.029583 17706 raft_consensus.cc:3055] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:52.068821 17706 raft_consensus.cc:740] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:52.034570 17720 raft_consensus.cc:493] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:52.034694 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:52.034916 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:52.069797 17706 consensus_queue.cc:260] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:52.070633 17059 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.071239 17706 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.035452 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:52.036244 17725 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:52.075855 17698 raft_consensus.cc:2749] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.037623 17058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.037822 17733 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:52.038089 17031 leader_election.cc:336] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.079221 17031 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:52.039433 17713 raft_consensus.cc:2749] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.039939 17734 raft_consensus.cc:2749] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.042760 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:52.043679 16987 raft_consensus.cc:2468] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:52.043836 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:52.044782 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:52.083422 17061 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.045888 17721 consensus_queue.cc:1059] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:52.085336 17701 raft_consensus.cc:3055] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:52.085749 17701 raft_consensus.cc:740] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
W20260709 13:05:52.086009 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 47966.8us
W20260709 13:05:52.086887 17031 leader_election.cc:336] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.087414 17031 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:52.088754 17031 consensus_peers.cc:597] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.029541 17714 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:52.090721 17028 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.091988 17713 raft_consensus.cc:2749] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.093315 17031 consensus_peers.cc:597] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:52.094220 17031 consensus_peers.cc:597] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:52.096688 17031 leader_election.cc:336] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.097136 17031 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:52.098198 17691 raft_consensus.cc:2749] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.100832 17059 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:52.104238 17734 raft_consensus.cc:2749] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.105515 16879 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:52.106459 17733 raft_consensus.cc:2804] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:52.106961 17733 raft_consensus.cc:697] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:52.097378 17061 raft_consensus.cc:2468] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:52.034122 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:52.112787 17056 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.069583 17720 raft_consensus.cc:515] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:52.119057 17720 leader_election.cc:290] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:52.036362 17062 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:52.119764 17062 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:52.063602 17102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 66408.3us
I20260709 13:05:52.107887 17733 consensus_queue.cc:237] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.126477 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:52.128470 17102 leader_election.cc:304] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.130110 17706 raft_consensus.cc:2749] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.130275 17102 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.131234 17706 raft_consensus.cc:2749] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.086529 17701 consensus_queue.cc:260] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:52.063771 17105 consensus_peers.cc:597] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.132339 17701 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:52.133603 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 69890.9us
W20260709 13:05:52.134312 17105 consensus_peers.cc:597] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.139094 17056 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:52.109539 17030 leader_election.cc:304] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:52.141359 16879 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:52.142367 17720 raft_consensus.cc:2804] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:52.143126 17720 raft_consensus.cc:697] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:52.144110 17102 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.143985 17720 consensus_queue.cc:237] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.144940 17698 raft_consensus.cc:2749] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.142326 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 45665.9us
W20260709 13:05:52.147818 17031 leader_election.cc:336] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:52.148290 17031 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:52.108405 17712 raft_consensus.cc:3060] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.151866 17734 raft_consensus.cc:2749] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.147223 17714 raft_consensus.cc:2749] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.153302 17059 raft_consensus.cc:1240] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:52.154444 17058 raft_consensus.cc:1240] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:52.155313 17725 raft_consensus.cc:493] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:52.155665 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.155727 17062 raft_consensus.cc:2468] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:52.155898 17725 raft_consensus.cc:515] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:52.158025 16879 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:52.159571 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.171576 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:52.182361 17031 consensus_peers.cc:597] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.157357 17736 raft_consensus.cc:2804] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:52.202603 17736 raft_consensus.cc:697] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:52.203538 17736 consensus_queue.cc:237] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:52.204890 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:52.206681 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4cf831311924bb7b8afbc30043e1550" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.172914 17733 raft_consensus.cc:2804] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:52.210067 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:52.218987 17733 raft_consensus.cc:697] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:52.219951 17733 consensus_queue.cc:237] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.210796 17130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3fbc4f00185436cbce1d8e24e6d1e3a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:52.206161 17031 consensus_peers.cc:597] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.211411 17065 raft_consensus.cc:1240] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:52.212203 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:52.214850 17454 consensus_queue.cc:1059] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:05:52.226883 17063 raft_consensus.cc:1240] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:52.195040 17725 leader_election.cc:290] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:52.195773 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:52.197727 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.190137 16879 leader_election.cc:304] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.215656 17102 leader_election.cc:304] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.236703 17454 raft_consensus.cc:2749] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.237331 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 47280.4us
I20260709 13:05:52.237926 17102 leader_election.cc:304] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.211784 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:52.233743 17031 consensus_peers.cc:597] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.239948 16879 leader_election.cc:304] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.240109 17454 raft_consensus.cc:2749] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.240953 17102 leader_election.cc:304] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.248867 16879 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.210935 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6703b86b3e4b4fad39ae2660fe6ff8" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.254240 17102 leader_election.cc:304] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.255074 17454 raft_consensus.cc:2749] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:52.251017 17105 consensus_peers.cc:597] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:52.213162 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:52.213796 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:52.212455 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:52.252108 16879 leader_election.cc:304] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.236044 17698 raft_consensus.cc:3055] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:52.238804 17719 raft_consensus.cc:2749] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.215320 17699 consensus_queue.cc:1059] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 13:05:52.252712 17692 raft_consensus.cc:2749] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.258808 17703 raft_consensus.cc:2749] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.253048 17735 raft_consensus.cc:2749] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.260510 17727 consensus_queue.cc:1059] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 13:05:52.282341 17727 raft_consensus.cc:3055] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:52.282836 17727 raft_consensus.cc:740] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:52.283689 17727 consensus_queue.cc:260] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.285128 17727 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.267181 16879 leader_election.cc:304] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.265508 17696 consensus_queue.cc:1059] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 13:05:52.265723 17698 raft_consensus.cc:740] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:52.297206 16879 leader_election.cc:304] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.299877 17454 raft_consensus.cc:3055] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:52.300418 17454 raft_consensus.cc:740] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:52.299962 16879 leader_election.cc:304] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.302994 17743 raft_consensus.cc:2749] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.305938 16879 leader_election.cc:304] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.305924 17454 consensus_queue.cc:260] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.316521 16879 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.293557 17720 raft_consensus.cc:2749] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.315945 16812 catalog_manager.cc:5697] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 1 to 2, leader changed from c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 2 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:52.320201 17454 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.305169 17719 raft_consensus.cc:2749] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.305064 17725 raft_consensus.cc:2749] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.322198 16879 leader_election.cc:304] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.320834 17703 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:05:52.324712 17728 raft_consensus.cc:2749] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.326901 16879 leader_election.cc:304] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.317703 17743 raft_consensus.cc:2749] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.343946 16879 leader_election.cc:304] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.347713 16879 leader_election.cc:304] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.358841 16879 leader_election.cc:304] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.360872 17743 raft_consensus.cc:2749] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.361311 16879 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.364619 16879 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.346428 17709 raft_consensus.cc:2749] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.350512 17703 raft_consensus.cc:2749] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.364044 17742 raft_consensus.cc:2749] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.367136 17735 raft_consensus.cc:2749] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.343195 17698 consensus_queue.cc:260] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.399108 17061 raft_consensus.cc:2468] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:52.399592 17698 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.388439 17743 raft_consensus.cc:2749] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.419270 17692 raft_consensus.cc:3055] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:52.419855 17692 raft_consensus.cc:740] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
W20260709 13:05:52.422329 16879 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 57825us
I20260709 13:05:52.423266 16879 leader_election.cc:304] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.424819 17725 raft_consensus.cc:2749] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.424567 16879 leader_election.cc:304] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.426092 17725 raft_consensus.cc:2749] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.427039 16879 leader_election.cc:304] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.427899 17725 raft_consensus.cc:2749] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.429095 16879 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.446197 16879 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:52.449644 17131 raft_consensus.cc:1275] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.421023 17692 consensus_queue.cc:260] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.452049 17692 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.454789 17725 raft_consensus.cc:2804] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:52.452988 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.455372 17725 raft_consensus.cc:697] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:52.456342 17725 consensus_queue.cc:237] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:52.442622 17720 raft_consensus.cc:2749] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:52.458465 17754 consensus_queue.cc:1048] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:52.459450 16879 leader_election.cc:304] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.462839 17755 raft_consensus.cc:2749] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.466926 16879 leader_election.cc:304] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.469324 17102 leader_election.cc:304] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.472715 17706 raft_consensus.cc:2804] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:52.473261 17728 raft_consensus.cc:2749] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.473307 17706 raft_consensus.cc:697] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:52.486713 17734 raft_consensus.cc:493] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:05:52.487569 17734 raft_consensus.cc:515] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:52.476579 17706 consensus_queue.cc:237] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:52.496687 17734 leader_election.cc:290] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:52.500813 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:52.506992 17061 raft_consensus.cc:1275] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.510457 16990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1adb90cc0554b10b2f20940047d6fc2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:52.521030 17030 leader_election.cc:304] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:52.524927 17744 consensus_queue.cc:1048] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 13:05:52.525039 17714 raft_consensus.cc:2749] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:52.539554 16986 raft_consensus.cc:2468] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:52.561297 17132 raft_consensus.cc:1275] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.565466 17728 consensus_queue.cc:1048] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:52.587960 17030 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:05:52.588922 17732 raft_consensus.cc:2804] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:52.589584 17732 raft_consensus.cc:697] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:52.590611 17732 consensus_queue.cc:237] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:52.595058 16811 catalog_manager.cc:5697] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:52.594756 16809 catalog_manager.cc:5697] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } }
I20260709 13:05:52.601956 16811 catalog_manager.cc:5697] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:52.608199 16811 catalog_manager.cc:5697] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:52.619485 17132 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:52.639642 17061 raft_consensus.cc:1275] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.644438 17725 consensus_queue.cc:1048] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:52.662848 17132 raft_consensus.cc:1275] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.671259 17714 consensus_queue.cc:1048] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:52.677997 17136 raft_consensus.cc:1275] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.680330 16987 raft_consensus.cc:1275] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.682417 17059 raft_consensus.cc:1275] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.681093 17060 raft_consensus.cc:1275] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.684795 17755 consensus_queue.cc:1048] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:05:52.695662 17728 consensus_queue.cc:1048] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:52.687136 17714 consensus_queue.cc:1048] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 13:05:52.709523 17720 consensus_queue.cc:1048] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:05:52.846946  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:05:52.848083  7481 tablet_replica.cc:333] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:52.849090  7481 raft_consensus.cc:2243] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:52.849440 17135 raft_consensus.cc:1275] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.849725  7481 raft_consensus.cc:2272] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:52.851569 17136 raft_consensus.cc:1275] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.854549  7481 tablet_replica.cc:333] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:52.855609  7481 raft_consensus.cc:2243] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:52.856210  7481 raft_consensus.cc:2272] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:52.858433 17755 consensus_queue.cc:1048] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:52.857623 17728 consensus_queue.cc:1048] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:52.886902  7481 tablet_replica.cc:333] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:52.888106  7481 raft_consensus.cc:2243] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:52.888741  7481 raft_consensus.cc:2272] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:52.941936  7481 tablet_replica.cc:333] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:52.944379 17725 raft_consensus.cc:493] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:52.945013 17725 raft_consensus.cc:515] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:52.945582  7481 raft_consensus.cc:2243] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:52.957610 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:52.958230 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3da49eeaa7457e8f3b1848d46bcea5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:52.959542 16809 catalog_manager.cc:5697] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 1 to 2, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:52.962510 16879 leader_election.cc:304] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:52.963572 17728 raft_consensus.cc:2749] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:52.964439 17725 leader_election.cc:290] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:52.957875  7481 raft_consensus.cc:2272] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:52.969852 17056 raft_consensus.cc:1275] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:52.969871  7481 tablet_replica.cc:333] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:52.974820  7481 raft_consensus.cc:2243] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:52.975414  7481 raft_consensus.cc:2272] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:52.978375 17755 consensus_queue.cc:1048] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:52.990713  7481 tablet_replica.cc:333] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.016482  7481 raft_consensus.cc:2243] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.017139  7481 raft_consensus.cc:2272] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.020308  7481 tablet_replica.cc:333] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.021338  7481 raft_consensus.cc:2243] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.022125  7481 raft_consensus.cc:2272] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.024259 16812 catalog_manager.cc:5697] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 1 to 2, leader changed from c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 2 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:53.030709 17131 raft_consensus.cc:3060] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:53.037148 17060 raft_consensus.cc:1275] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:53.046985 17724 consensus_queue.cc:1048] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:53.064604  7481 tablet_replica.cc:333] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.065893  7481 raft_consensus.cc:2243] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.066576  7481 raft_consensus.cc:2272] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.070400 16811 catalog_manager.cc:5697] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } }
I20260709 13:05:53.078883  7481 tablet_replica.cc:333] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.080142  7481 raft_consensus.cc:2243] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.080835  7481 raft_consensus.cc:2272] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.089336  7481 tablet_replica.cc:333] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.091149  7481 raft_consensus.cc:2243] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.104243  7481 raft_consensus.cc:2272] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.107044  7481 tablet_replica.cc:333] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.108040  7481 raft_consensus.cc:2243] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.112907  7481 raft_consensus.cc:2272] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.115135 17131 raft_consensus.cc:1275] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:53.116428 16987 raft_consensus.cc:1275] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:53.117851  7481 tablet_replica.cc:333] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.120498 16990 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:53.118100 17697 consensus_queue.cc:1048] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 13:05:53.124406  7481 raft_consensus.cc:2243] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.124780 17734 consensus_queue.cc:1048] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:53.141332  7481 raft_consensus.cc:2272] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.145586 16990 raft_consensus.cc:1275] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:53.161965 17724 consensus_queue.cc:1048] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 13:05:53.149088  7481 tablet_replica.cc:333] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.180966  7481 raft_consensus.cc:2243] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.181492  7481 raft_consensus.cc:2272] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.183995  7481 tablet_replica.cc:333] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.184805  7481 raft_consensus.cc:2243] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.185289  7481 raft_consensus.cc:2272] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.196210  7481 tablet_replica.cc:333] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.197191  7481 raft_consensus.cc:2243] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.198325  7481 raft_consensus.cc:2272] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.268882 17755 raft_consensus.cc:493] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:53.269208 17728 raft_consensus.cc:493] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:05:53.269455 17755 raft_consensus.cc:515] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:53.269946 17728 raft_consensus.cc:515] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:53.271795 17755 leader_election.cc:290] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:53.273080 17728 leader_election.cc:290] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:53.275274  7481 tablet_replica.cc:333] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.277210 17699 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:53.277781 17699 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.280372 17699 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:53.280871  7481 raft_consensus.cc:2243] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.281610  7481 raft_consensus.cc:2272] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.283735 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:53.284510 16989 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:53.285419 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:53.286182 17060 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:53.287739 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:53.289114 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:53.289613  7481 tablet_replica.cc:333] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.290369 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:53.290592 17102 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:53.292292 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:53.294029 17724 raft_consensus.cc:2804] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:53.294507 17724 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:53.290712  7481 raft_consensus.cc:2243] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.294878 17724 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:53.295387  7481 raft_consensus.cc:2272] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.297778 16879 leader_election.cc:304] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:53.299278  7481 tablet_replica.cc:333] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.300300  7481 raft_consensus.cc:2243] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.306746 17720 raft_consensus.cc:2749] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:53.308130 16879 leader_election.cc:304] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:53.308815 17720 raft_consensus.cc:2749] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:53.305881 17724 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.317749  7481 raft_consensus.cc:2272] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.319644 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:53.320322 16986 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:53.321499 17724 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:53.327135 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:53.327822 17056 raft_consensus.cc:3060] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:53.327872 16986 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 3.
I20260709 13:05:53.330410 17104 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:53.331681 17699 raft_consensus.cc:2804] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:53.332165 17699 raft_consensus.cc:697] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:53.332970 17699 consensus_queue.cc:237] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.337477  7481 tablet_replica.cc:333] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.338445  7481 raft_consensus.cc:2243] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.339006  7481 raft_consensus.cc:2272] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.341857  7481 tablet_replica.cc:333] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.342685  7481 raft_consensus.cc:2243] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.343217  7481 raft_consensus.cc:2272] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.345782 17056 raft_consensus.cc:2468] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 3.
I20260709 13:05:53.346114  7481 tablet_replica.cc:333] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.345504 16811 catalog_manager.cc:5697] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:53.347013  7481 raft_consensus.cc:2243] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.347589  7481 raft_consensus.cc:2272] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.350507  7481 tablet_replica.cc:333] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.351387  7481 raft_consensus.cc:2243] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.351940  7481 raft_consensus.cc:2272] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.355252  7481 tablet_replica.cc:333] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.356137  7481 raft_consensus.cc:2243] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.356670  7481 raft_consensus.cc:2272] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.359562  7481 tablet_replica.cc:333] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.360411  7481 raft_consensus.cc:2243] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.361045  7481 raft_consensus.cc:2272] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.363792  7481 tablet_replica.cc:333] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.364641  7481 raft_consensus.cc:2243] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.365150  7481 raft_consensus.cc:2272] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.367785  7481 tablet_replica.cc:333] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.368613  7481 raft_consensus.cc:2243] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.369230  7481 raft_consensus.cc:2272] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.372251  7481 tablet_replica.cc:333] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.373072  7481 raft_consensus.cc:2243] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.374142  7481 raft_consensus.cc:2272] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.376811  7481 tablet_replica.cc:333] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.377647  7481 raft_consensus.cc:2243] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.378731  7481 raft_consensus.cc:2272] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.397559  7481 tablet_replica.cc:333] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.398559  7481 raft_consensus.cc:2243] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.399151  7481 raft_consensus.cc:2272] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.411237  7481 tablet_replica.cc:333] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.412371  7481 raft_consensus.cc:2243] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.413867  7481 raft_consensus.cc:2272] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.420717  7481 tablet_replica.cc:333] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.421813  7481 raft_consensus.cc:2243] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.422457  7481 raft_consensus.cc:2272] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.427911  7481 tablet_replica.cc:333] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.428936  7481 raft_consensus.cc:2243] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.429762  7481 raft_consensus.cc:2272] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.432966  7481 tablet_replica.cc:333] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.433876  7481 raft_consensus.cc:2243] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.435135  7481 raft_consensus.cc:2272] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.441480  7481 tablet_replica.cc:333] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.442408  7481 raft_consensus.cc:2243] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.443619  7481 raft_consensus.cc:2272] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.448096  7481 tablet_replica.cc:333] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.449092  7481 raft_consensus.cc:2243] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.450366  7481 raft_consensus.cc:2272] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.453882  7481 tablet_replica.cc:333] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.454811  7481 raft_consensus.cc:2243] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.455331  7481 raft_consensus.cc:2272] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.458081  7481 tablet_replica.cc:333] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.458974  7481 raft_consensus.cc:2243] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.459610  7481 raft_consensus.cc:2272] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.462924  7481 tablet_replica.cc:333] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.463842  7481 raft_consensus.cc:2243] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.464469  7481 raft_consensus.cc:2272] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.467898  7481 tablet_replica.cc:333] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.468771  7481 raft_consensus.cc:2243] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.470041  7481 raft_consensus.cc:2272] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.473412  7481 tablet_replica.cc:333] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.474495  7481 raft_consensus.cc:2243] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.475132  7481 raft_consensus.cc:2272] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.478608  7481 tablet_replica.cc:333] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.479573  7481 raft_consensus.cc:2243] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.480170  7481 raft_consensus.cc:2272] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.484616  7481 tablet_replica.cc:333] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.485530  7481 raft_consensus.cc:2243] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.486086  7481 raft_consensus.cc:2272] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.489192  7481 tablet_replica.cc:333] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.490168  7481 raft_consensus.cc:2243] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:05:53.491354  7481 raft_consensus.cc:2272] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.494621  7481 tablet_replica.cc:333] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.495434  7481 raft_consensus.cc:2243] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.496073  7481 raft_consensus.cc:2272] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.499197  7481 tablet_replica.cc:333] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: stopping tablet replica
I20260709 13:05:53.499953  7481 raft_consensus.cc:2243] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:53.500626  7481 raft_consensus.cc:2272] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:53.543792 17699 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:53.544466 17699 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.546849 17699 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:53.547768 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:53.548542 16989 raft_consensus.cc:2468] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:53.550700 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:53.551728 17060 raft_consensus.cc:2393] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 2: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:53.553268 17104 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:53.555932 17699 raft_consensus.cc:2804] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:53.556342 17699 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:53.556730 17699 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:53.590402 17699 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.593906 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:53.602355 16986 raft_consensus.cc:2468] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:53.603613 17104 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:53.604736 17727 raft_consensus.cc:2804] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:53.628805 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:53.630343 17056 raft_consensus.cc:2393] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 2: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:53.630857 17699 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:53.632894 17727 raft_consensus.cc:697] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:53.643720 17727 consensus_queue.cc:237] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.665489 16811 catalog_manager.cc:5697] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 1 to 2, leader changed from c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66) to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:53.752286 16986 raft_consensus.cc:1275] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:05:53.753906 17727 consensus_queue.cc:1048] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:05:53.778374 16957 consensus_peers.cc:597] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:53.811715 17056 raft_consensus.cc:1275] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:05:53.820225 17785 consensus_queue.cc:1048] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:53.821013 17754 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:53.822013 17754 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.824366 17754 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:53.826483 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:53.827203 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:53.827883 17135 raft_consensus.cc:2468] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:53.829255 16954 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:53.830010 17754 raft_consensus.cc:2804] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:53.830376 17754 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:53.827220 17061 raft_consensus.cc:2468] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 2.
I20260709 13:05:53.830811 17754 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:53.846711 17754 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.852372 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:53.853178 17056 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:53.868916 16957 consensus_peers.cc:597] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:53.871587 17031 consensus_peers.cc:597] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:53.892794 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:53.893549 17135 raft_consensus.cc:3060] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:05:53.895852 17056 raft_consensus.cc:2468] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 3.
I20260709 13:05:53.897203 16954 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:53.898236 17701 raft_consensus.cc:2804] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:53.899812 17701 raft_consensus.cc:697] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 3 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:53.900811 17701 consensus_queue.cc:237] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:53.905985 17135 raft_consensus.cc:2468] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 3.
I20260709 13:05:53.906029 17754 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:53.925894 16811 catalog_manager.cc:5697] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:54.034857 17734 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:54.035540 17734 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:54.038576 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:54.039393 16989 raft_consensus.cc:2468] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:54.040769 17030 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.040967 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.041729 17732 raft_consensus.cc:2804] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:54.042107 17732 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:54.042459 17732 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:54.039454 17734 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:54.051683 16957 consensus_peers.cc:597] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.056696 17732 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:54.065389 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:54.072242 16989 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:54.070824 17105 consensus_peers.cc:597] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.063730 16957 consensus_peers.cc:597] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.084278 16957 consensus_peers.cc:597] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.088848 16989 raft_consensus.cc:2468] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 3.
I20260709 13:05:54.090236 17732 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:05:54.106365 17105 consensus_peers.cc:597] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.118237 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:54.119680 17030 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.121464 17732 raft_consensus.cc:2804] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:54.121963 17732 raft_consensus.cc:697] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:54.122926 17744 raft_consensus.cc:493] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.123459 17744 raft_consensus.cc:515] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:54.122843 17732 consensus_queue.cc:237] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
W20260709 13:05:54.136005 16957 leader_election.cc:336] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.136924 17744 leader_election.cc:290] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:54.140205 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.145886 16954 leader_election.cc:304] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:54.147543 17744 raft_consensus.cc:2749] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:54.144469 16809 catalog_manager.cc:5697] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 2 to 3, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 3 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:54.163146 17105 consensus_peers.cc:597] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.164000 16957 consensus_peers.cc:597] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.174923 16957 consensus_peers.cc:597] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.177817 17060 raft_consensus.cc:1240] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 2. Current term is 3. Ops: []
I20260709 13:05:54.179356 17785 consensus_queue.cc:1059] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:54.180032 17785 consensus_queue.cc:1243] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer f83f5ad22c2547b29c99ae897ea9b61b log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:05:54.180485 17727 raft_consensus.cc:3055] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:05:54.180822 17727 raft_consensus.cc:740] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:54.181825 17727 consensus_queue.cc:260] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:05:54.183000 17727 raft_consensus.cc:3060] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:54.208602 17105 consensus_peers.cc:597] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.222661 17785 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.223212 17785 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.225034 17785 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.225911 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:54.236295 17031 consensus_peers.cc:597] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.253273 17732 raft_consensus.cc:493] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
W20260709 13:05:54.253602 17105 leader_election.cc:336] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.253993 17732 raft_consensus.cc:515] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:54.254257 17105 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:54.255122 17785 raft_consensus.cc:2749] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:54.258246 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:54.258945 16989 raft_consensus.cc:2468] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:54.260206 17030 leader_election.cc:304] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.260980 17797 raft_consensus.cc:2804] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:54.261375 17797 raft_consensus.cc:493] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.261749 17797 raft_consensus.cc:3060] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.262277 17732 leader_election.cc:290] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
W20260709 13:05:54.262951 17031 leader_election.cc:336] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.270212 17797 raft_consensus.cc:515] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:54.274756 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa31fcff86ca4af6b0cb7c4be6736c57" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:54.275372 16989 raft_consensus.cc:3060] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:54.279018 17031 leader_election.cc:336] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.281239 17732 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.281783 17732 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.284626 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.285229 17135 raft_consensus.cc:2468] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 1.
I20260709 13:05:54.287101 17732 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.288590 17028 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.289492 17732 raft_consensus.cc:2804] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:54.289904 17732 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.292423 17732 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.297041 16989 raft_consensus.cc:2468] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
W20260709 13:05:54.271595 17105 consensus_peers.cc:597] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.299130 16957 consensus_peers.cc:597] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.300990 17030 leader_election.cc:304] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.302855 17734 raft_consensus.cc:2804] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:54.306384 17734 raft_consensus.cc:697] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
W20260709 13:05:54.303354 17031 leader_election.cc:336] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.311892 17734 consensus_queue.cc:237] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:54.318701 17797 leader_election.cc:290] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
W20260709 13:05:54.320766 17105 consensus_peers.cc:597] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.327888 16957 consensus_peers.cc:597] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.331490 17732 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.336570 17105 consensus_peers.cc:597] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.343014 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:54.343827 17131 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:54.352427 17031 consensus_peers.cc:597] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.343638 16809 catalog_manager.cc:5697] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 1 to 2, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 2 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } }
W20260709 13:05:54.361476 17031 leader_election.cc:336] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
W20260709 13:05:54.369694 17031 consensus_peers.cc:597] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.374359 17131 raft_consensus.cc:2468] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:54.373046 17732 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.376225 17028 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.377592 17732 raft_consensus.cc:2804] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:54.378422 17732 raft_consensus.cc:697] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:54.380928 17732 consensus_queue.cc:237] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.389636 17031 consensus_peers.cc:597] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.390800 17105 consensus_peers.cc:597] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.391743 17785 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.392354 17785 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.395563 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.398272 17785 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.405593 17785 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.406425 17785 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.411670 17797 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.412675 17797 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.417367 17797 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.419626 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.420445 17056 raft_consensus.cc:1275] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:05:54.421455 17802 raft_consensus.cc:493] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.422114 17802 raft_consensus.cc:515] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:54.424201 17802 leader_election.cc:290] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:54.424640 17744 consensus_queue.cc:1048] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:05:54.425009 17031 leader_election.cc:336] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.425549 17031 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.427750 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:54.428937 17800 raft_consensus.cc:2749] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:05:54.436597 17031 consensus_peers.cc:597] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.439508 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
W20260709 13:05:54.442106 17105 leader_election.cc:336] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.443866 17105 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:54.445384 17802 raft_consensus.cc:2749] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:54.443292 17785 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:54.452560 17105 leader_election.cc:336] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.453209 17105 leader_election.cc:304] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:54.454200 17131 raft_consensus.cc:1275] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:05:54.454509 17727 raft_consensus.cc:2749] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:54.455001 17734 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.455575 17734 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:54.457758 17734 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:54.459444 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:54.464479 17754 consensus_queue.cc:1048] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:54.485901 17800 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.486460 17800 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.489264 17802 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.490342 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
W20260709 13:05:54.491581 17031 leader_election.cc:336] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.492200 17031 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:54.492767 17802 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.497987 17802 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.499440 17797 raft_consensus.cc:2749] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:54.501909 17131 raft_consensus.cc:2468] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:54.504035 17028 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.505512 17798 raft_consensus.cc:2804] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:54.505939 17798 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.506295 17798 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:54.506929 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.518040 17800 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:54.524433 17031 consensus_peers.cc:597] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.530105 17031 consensus_peers.cc:597] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.536034 17031 consensus_peers.cc:597] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.454396 17105 leader_election.cc:336] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.541858 17105 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:54.542930 17802 raft_consensus.cc:2749] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:05:54.543081 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 88719.4us
I20260709 13:05:54.528746 17798 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.547828 17056 raft_consensus.cc:2393] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40c8156681dd4a7bb2d037afbeb01580 in current term 3: Already voted for candidate f83f5ad22c2547b29c99ae897ea9b61b in this term.
I20260709 13:05:54.552459 17785 raft_consensus.cc:493] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
W20260709 13:05:54.552824 17031 leader_election.cc:336] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.553118 17785 raft_consensus.cc:515] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:54.555514 17785 leader_election.cc:290] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:54.552491 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:54.556262 17131 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:54.557296 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:54.556957 17798 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:54.565537 17105 consensus_peers.cc:597] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.566155 17105 leader_election.cc:336] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.567097 17805 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.572208 17105 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:54.572378 17805 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.573311 17105 consensus_peers.cc:597] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.574702 17805 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.577008 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.577340 17132 raft_consensus.cc:1275] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:05:54.577749 17132 pending_rounds.cc:85] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: Aborting all ops after (but not including) 1
I20260709 13:05:54.577982 17132 pending_rounds.cc:107] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:05:54.578527 17132 raft_consensus.cc:2889] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:05:54.580304 17732 consensus_queue.cc:1048] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:54.585187 17785 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.585724 17785 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.586318 17105 leader_election.cc:336] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.588325 17105 leader_election.cc:304] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:54.589213 17805 raft_consensus.cc:2749] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:54.589319 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.590188 17060 raft_consensus.cc:2468] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:54.591547 17102 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.592717 17805 raft_consensus.cc:2804] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:54.593175 17805 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.588032 17785 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.593698 17805 raft_consensus.cc:3060] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.546723 16809 catalog_manager.cc:5697] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 1 to 2, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 2 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:54.571619 17031 leader_election.cc:336] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.599124 16989 raft_consensus.cc:1275] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:05:54.600953 17732 consensus_queue.cc:1048] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:54.608700 17131 raft_consensus.cc:2468] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 3.
I20260709 13:05:54.612617 17802 raft_consensus.cc:2749] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:05:54.616130 17031 consensus_peers.cc:597] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.623533 17105 leader_election.cc:336] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.624136 17105 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:54.627427 17802 raft_consensus.cc:2749] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:54.631688 17754 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.632402 17754 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:54.634320 16957 consensus_peers.cc:597] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.635177 17732 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.636080 17754 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:54.635766 17732 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.640050 16957 leader_election.cc:336] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.640724 17028 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.641249 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.641933 17131 raft_consensus.cc:2468] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:54.642637 17797 raft_consensus.cc:2804] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:05:54.645328 17031 leader_election.cc:336] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.645665 17797 raft_consensus.cc:697] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:54.646270 17732 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.646600 17797 consensus_queue.cc:237] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.620904 17805 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.648733 17727 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.649147 17805 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.649307 17727 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.651887 17105 consensus_peers.cc:597] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:54.652740 17105 leader_election.cc:336] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.653360 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:54.654305 17056 raft_consensus.cc:3060] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.654347 17727 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:54.656606  7481 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 16937)
I20260709 13:05:54.655745 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.672468 17800 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.673208 17028 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.673081 17800 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.675015 17105 consensus_peers.cc:597] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.677369 17797 raft_consensus.cc:2804] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:54.678633 17797 raft_consensus.cc:493] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.679001 17797 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:54.685523 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 66275.6us
I20260709 13:05:54.694041 17800 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.682915 16811 catalog_manager.cc:5697] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 2 to 3, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 3 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:54.702692 17797 raft_consensus.cc:515] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.709620 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.711025 17797 leader_election.cc:290] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:05:54.713636 17105 leader_election.cc:336] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.713855 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71be301ca5434f9b8880e7a03ef1ebca" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
W20260709 13:05:54.714356 17105 leader_election.cc:336] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.714748 17132 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:54.715616 17031 leader_election.cc:336] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.716099 17031 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.720465 17701 raft_consensus.cc:493] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.721180 16987 raft_consensus.cc:1275] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:54.721016 17701 raft_consensus.cc:515] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:54.724726 17102 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:54.719803 17797 raft_consensus.cc:2749] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:54.725926 17785 raft_consensus.cc:2749] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:54.734067 17701 leader_election.cc:290] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.735503 17798 consensus_queue.cc:1048] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:05:54.741679 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 46060.2us
W20260709 13:05:54.742365 17031 consensus_peers.cc:597] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:54.745450 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.746168 17063 raft_consensus.cc:2468] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
W20260709 13:05:54.742874 17031 leader_election.cc:336] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.749217 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.749890 17135 raft_consensus.cc:2468] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:54.741595 17132 raft_consensus.cc:2468] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 3.
I20260709 13:05:54.752557 16954 leader_election.cc:304] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:54.754124 17813 raft_consensus.cc:2804] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:54.754499 17813 raft_consensus.cc:493] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.754828 17813 raft_consensus.cc:3060] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.754702 17028 leader_election.cc:304] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.758147 17798 raft_consensus.cc:2804] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:54.758651 17798 raft_consensus.cc:697] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:54.759479 17798 consensus_queue.cc:237] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.761051 16957 leader_election.cc:336] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.762207 16954 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.765347 17701 raft_consensus.cc:2804] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:54.769421 17701 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.769908 17701 raft_consensus.cc:3060] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.771739 17802 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.772251 17802 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.775218 17802 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.776791 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:54.777624 17057 raft_consensus.cc:2468] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 1.
I20260709 13:05:54.778740 17102 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.780262 17802 raft_consensus.cc:2804] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:54.780627 17802 raft_consensus.cc:493] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.780880 17802 raft_consensus.cc:3060] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:54.784214 17031 consensus_peers.cc:597] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:54.794328 17813 raft_consensus.cc:515] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
W20260709 13:05:54.803004 17105 leader_election.cc:336] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.806282 17797 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.806991 17797 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.806556 17813 leader_election.cc:290] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.809239 17797 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.810181 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5eff39a6814f6e8a37a48064b3dd2e" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:54.811064 17132 raft_consensus.cc:3060] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.811780 17701 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:54.814881 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.815691 17131 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:54.817098 17028 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.817997 17816 raft_consensus.cc:2804] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:54.818405 17816 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.818792 17816 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:54.821661 16957 leader_election.cc:336] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.825187 17802 raft_consensus.cc:515] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.829154 17056 raft_consensus.cc:2468] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
I20260709 13:05:54.828681 17057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "277a6b1c93854935be35e234b98330d2" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:54.832737 17057 raft_consensus.cc:3060] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:54.835829 17797 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.836596 17797 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.838261 16957 leader_election.cc:336] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.843091 17701 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:54.844837 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:54.846525 17063 raft_consensus.cc:3060] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:54.848049 17031 leader_election.cc:336] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.854928 17063 raft_consensus.cc:2468] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 2.
I20260709 13:05:54.856335 16954 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.856645 17102 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.857499 17813 raft_consensus.cc:2804] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:54.857889 17727 raft_consensus.cc:2804] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:54.858006 17813 raft_consensus.cc:697] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:54.858541 17727 raft_consensus.cc:697] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:54.858994 17813 consensus_queue.cc:237] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:54.859496 17727 consensus_queue.cc:237] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.860085 17132 raft_consensus.cc:2468] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 2.
I20260709 13:05:54.860561 16811 catalog_manager.cc:5697] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 2 to 3, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 3 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:54.862354 17797 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.866356 16954 leader_election.cc:304] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.859544 17816 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.868395 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.870097 17812 raft_consensus.cc:2804] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:05:54.875972 17031 leader_election.cc:336] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.881305 16811 catalog_manager.cc:5697] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 1 to 2, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:54.882488 17802 leader_election.cc:290] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.882355 17132 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:54.883320 17816 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.885829 17057 raft_consensus.cc:2468] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 2.
W20260709 13:05:54.888022 17105 leader_election.cc:336] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.885633 17797 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.882441 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:54.899331 17131 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:54.899485 17028 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.899624 17102 leader_election.cc:304] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.900707 17816 raft_consensus.cc:2804] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:54.900758 17802 raft_consensus.cc:2804] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:54.901268 17816 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.887014 17812 raft_consensus.cc:697] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:54.902070 17816 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:54.903096 17797 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.905319 17701 raft_consensus.cc:493] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.905864 17701 raft_consensus.cc:515] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:54.907034 17797 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.901566 17802 raft_consensus.cc:697] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:54.907701 17701 leader_election.cc:290] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.902864 17812 consensus_queue.cc:237] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:54.908195 17802 consensus_queue.cc:237] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:54.897547 17031 leader_election.cc:336] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.895972 16809 catalog_manager.cc:5697] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 1 to 2, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:54.911660 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.912424 17135 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:54.913780 17028 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:54.914637 17798 raft_consensus.cc:2804] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:54.915017 17798 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.915395 17798 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:54.917936 17031 leader_election.cc:336] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.911042 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:54.923867 17136 raft_consensus.cc:2468] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 1.
I20260709 13:05:54.925981 16954 leader_election.cc:304] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:54.928185 17754 raft_consensus.cc:2804] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:54.928694 17754 raft_consensus.cc:493] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:05:54.929143 17754 raft_consensus.cc:3060] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:54.929883 16957 leader_election.cc:336] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.950256 17816 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.952546 17816 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.953481 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:54.955662 17136 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:54.963779 16809 catalog_manager.cc:5697] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 1 to 2, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } }
W20260709 13:05:54.965668 17031 leader_election.cc:336] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.922549 17131 raft_consensus.cc:2468] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 3.
I20260709 13:05:54.980145 17136 raft_consensus.cc:2468] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 3.
I20260709 13:05:54.981544 17028 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:54.982098 17798 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.977357 17754 raft_consensus.cc:515] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:54.982659 17816 raft_consensus.cc:2804] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:54.988729 17816 raft_consensus.cc:697] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:54.991321 17132 raft_consensus.cc:1275] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:54.993418 17797 consensus_queue.cc:1048] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:05:54.994014 16957 leader_election.cc:336] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.993743 17816 consensus_queue.cc:237] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:54.995469 16809 catalog_manager.cc:5697] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 1 to 2, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 2 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:54.997534 17132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:54.994477 17798 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:54.998297 17132 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:55.000177 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af54823c7c644c92b460fde468cdfaa0" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:05:55.000828 17136 raft_consensus.cc:3060] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:55.008136 17028 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
W20260709 13:05:55.009610 17031 leader_election.cc:336] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:05:54.991022 17754 leader_election.cc:290] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:55.013536 17798 raft_consensus.cc:2804] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:55.016098 17798 raft_consensus.cc:697] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:55.016966 17798 consensus_queue.cc:237] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:55.039029 17031 consensus_peers.cc:597] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.043048 17136 raft_consensus.cc:2468] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 2.
I20260709 13:05:55.043522 17132 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 3.
I20260709 13:05:55.047183 17028 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:55.048504 17816 raft_consensus.cc:2804] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:55.049064 17816 raft_consensus.cc:697] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Becoming Leader. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:05:55.050393 17816 consensus_queue.cc:237] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:55.087345 16954 leader_election.cc:304] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531
I20260709 13:05:55.114997 17132 raft_consensus.cc:1275] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:05:55.122028 17754 raft_consensus.cc:2804] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:05:55.124060 17732 consensus_queue.cc:1048] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:05:55.129033 17754 raft_consensus.cc:697] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:55.131595 17754 consensus_queue.cc:237] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:55.103250 16809 catalog_manager.cc:5697] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 2 to 3, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 3 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:55.157133 17031 consensus_peers.cc:597] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.201138 17131 raft_consensus.cc:1275] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:05:55.203557 17816 consensus_queue.cc:1048] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:05:55.245577 17031 consensus_peers.cc:597] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.248225 16811 catalog_manager.cc:5697] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 2 to 3, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 3 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:55.250823 16811 catalog_manager.cc:5697] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b reported cstate change: term changed from 2 to 3, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67). New cstate: current_term: 3 leader_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:55.295514 17056 raft_consensus.cc:1275] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:55.299813 16809 catalog_manager.cc:5697] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 1 to 2, leader changed from 71a10d4adec64d58ba645d68425ed531 (127.7.78.65) to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:55.303324 17754 consensus_queue.cc:1048] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:05:55.331409 17131 raft_consensus.cc:1275] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:05:55.335408 16957 consensus_peers.cc:597] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.336097 17814 consensus_queue.cc:1048] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:05:55.345656 17105 consensus_peers.cc:597] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:55.367329 16957 consensus_peers.cc:597] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:55.420591 17105 consensus_peers.cc:597] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.432117 17056 raft_consensus.cc:1275] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:55.434278 17828 consensus_queue.cc:1048] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:55.538179 17056 raft_consensus.cc:1275] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:05:55.541456 17031 consensus_peers.cc:597] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.542850 17828 consensus_queue.cc:1048] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:55.555050 17132 raft_consensus.cc:1275] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:05:55.556938 17754 consensus_queue.cc:1048] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:05:55.602216 16957 consensus_peers.cc:597] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.622273 17132 raft_consensus.cc:1275] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:05:55.624342 17816 consensus_queue.cc:1048] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:05:55.628305 17136 raft_consensus.cc:1275] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:05:55.644869 17816 consensus_queue.cc:1048] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 13:05:55.663009  7481 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 16937)
W20260709 13:05:55.689843 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 75656.6us
W20260709 13:05:55.700874 17031 consensus_peers.cc:597] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.711306 17132 raft_consensus.cc:1275] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer f83f5ad22c2547b29c99ae897ea9b61b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:05:55.713881 17797 consensus_queue.cc:1048] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:05:55.759039 17031 consensus_peers.cc:597] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:55.830524  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
I20260709 13:05:56.062889  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:05:56.071696 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:56.083876 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:05:56.087222 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:05:56.089987  7481 server_base.cc:1061] running on GCE node
I20260709 13:05:56.095695  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:05:56.096017  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:05:56.096277  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602356096257 us; error 0 us; skew 500 ppm
I20260709 13:05:56.098054  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:05:56.101827  7481 webserver.cc:533] Webserver started at http://127.7.78.65:33595/ using document root <none> and password file <none>
I20260709 13:05:56.102416  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:05:56.102646  7481 fs_manager.cc:365] Using existing metadata directory in first data directory
W20260709 13:05:56.109068 16957 consensus_peers.cc:597] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.116629  7481 fs_manager.cc:714] Time spent opening directory manager: real 0.013s	user 0.002s	sys 0.009s
I20260709 13:05:56.127542 17849 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:05:56.128718  7481 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.004s
I20260709 13:05:56.129136  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "71a10d4adec64d58ba645d68425ed531"
format_stamp: "Formatted at 2026-07-09 13:05:34 on dist-test-slave-50cs"
I20260709 13:05:56.129719  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:05:56.175828  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:05:56.177222  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:05:56.179127  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:05:56.186524 17856 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
W20260709 13:05:56.234696 17105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111) [suppressed 411 similar messages]
W20260709 13:05:56.385516 16957 consensus_peers.cc:597] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:56.431638 16957 consensus_peers.cc:597] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:56.533558 17105 consensus_peers.cc:597] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.540343  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 13:05:56.540786  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.356s	user 0.006s	sys 0.005s
I20260709 13:05:56.541201  7481 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
W20260709 13:05:56.545629 17105 consensus_peers.cc:597] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.549167 17856 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:56.567641 16957 consensus_peers.cc:597] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.580421 17856 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:56.581578 17856 tablet_bootstrap.cc:492] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:56.582324 17856 ts_tablet_manager.cc:1403] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.033s	user 0.022s	sys 0.008s
I20260709 13:05:56.585294 17856 raft_consensus.cc:359] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:56.586072 17856 raft_consensus.cc:740] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:56.586845 17856 consensus_queue.cc:260] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:56.589674 17856 ts_tablet_manager.cc:1434] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:05:56.590581 17856 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:56.619746 16957 consensus_peers.cc:597] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:56.621266 17105 consensus_peers.cc:597] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.632867 17856 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:56.634063 17856 tablet_bootstrap.cc:492] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:56.634841 17856 ts_tablet_manager.cc:1403] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.044s	user 0.037s	sys 0.004s
I20260709 13:05:56.638249 17856 raft_consensus.cc:359] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:56.639102 17856 raft_consensus.cc:740] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:56.639978 17856 consensus_queue.cc:260] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:56.642370 17856 ts_tablet_manager.cc:1434] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:05:56.643278 17856 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:56.686033 17856 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:56.687121 17856 tablet_bootstrap.cc:492] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:56.687808 17856 ts_tablet_manager.cc:1403] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.045s	user 0.022s	sys 0.004s
W20260709 13:05:56.689688 17031 consensus_peers.cc:597] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.690601 17856 raft_consensus.cc:359] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:56.691460 17856 raft_consensus.cc:740] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:56.692221 17856 consensus_queue.cc:260] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:56.697491 17856 ts_tablet_manager.cc:1434] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 13:05:56.698431 17856 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:56.721458 16957 consensus_peers.cc:597] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:56.729180 17105 consensus_peers.cc:597] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.734481 17856 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:56.735772 17856 tablet_bootstrap.cc:492] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:56.736407 17856 ts_tablet_manager.cc:1403] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.038s	user 0.017s	sys 0.000s
I20260709 13:05:56.738961 17856 raft_consensus.cc:359] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:56.739634 17856 raft_consensus.cc:740] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:56.740429 17856 consensus_queue.cc:260] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:56.740882 17105 consensus_peers.cc:597] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.742780 17856 ts_tablet_manager.cc:1434] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:05:56.743593 17856 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:56.766017 17856 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:56.767282 17856 tablet_bootstrap.cc:492] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:56.768049 17856 ts_tablet_manager.cc:1403] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.008s
W20260709 13:05:56.768980 17031 consensus_peers.cc:597] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:56.775635 17031 consensus_peers.cc:597] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:56.776857 17856 raft_consensus.cc:359] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:56.777748 17856 raft_consensus.cc:740] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:56.778539 17856 consensus_queue.cc:260] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:56.780874 17856 ts_tablet_manager.cc:1434] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I20260709 13:05:56.781841 17856 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:56.863777 17856 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:56.864739 17856 tablet_bootstrap.cc:492] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:56.865448 17856 ts_tablet_manager.cc:1403] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.084s	user 0.012s	sys 0.003s
I20260709 13:05:56.867946 17856 raft_consensus.cc:359] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:56.868671 17856 raft_consensus.cc:740] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:56.869387 17856 consensus_queue.cc:260] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:56.883657 17856 ts_tablet_manager.cc:1434] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 13:05:56.884604 17856 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:56.907146  7481 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 13:05:56.907578  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.366s	user 0.300s	sys 0.012s
I20260709 13:05:56.919668 17856 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:56.920765 17856 tablet_bootstrap.cc:492] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:56.921530 17856 ts_tablet_manager.cc:1403] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.037s	user 0.017s	sys 0.011s
I20260709 13:05:56.924266 17856 raft_consensus.cc:359] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:56.924993 17856 raft_consensus.cc:740] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:56.925782 17856 consensus_queue.cc:260] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:56.928160 17856 ts_tablet_manager.cc:1434] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:05:56.928922 17856 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:57.019829 17105 consensus_peers.cc:597] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.020714 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 47415.8us
W20260709 13:05:57.066433 17031 consensus_peers.cc:597] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.112048 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 86858us
W20260709 13:05:57.124639 17105 consensus_peers.cc:597] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.125555 17102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 77702us
W20260709 13:05:57.127802 17105 consensus_peers.cc:597] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.128367 17105 consensus_peers.cc:597] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.139343 17105 consensus_peers.cc:597] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.178423 16957 consensus_peers.cc:597] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.195752 16957 consensus_peers.cc:597] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.194449 17031 consensus_peers.cc:597] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.197865 17031 consensus_peers.cc:597] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.199731 17031 consensus_peers.cc:597] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.222978 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 77954.8us
W20260709 13:05:57.222144 16957 consensus_peers.cc:597] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.230047 16957 consensus_peers.cc:597] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.253486 17104 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 56700.2us
W20260709 13:05:57.258220 17031 consensus_peers.cc:597] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:57.263896 17856 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.285715 17856 tablet_bootstrap.cc:492] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.286542 17856 ts_tablet_manager.cc:1403] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.358s	user 0.102s	sys 0.137s
W20260709 13:05:57.287381 17105 consensus_peers.cc:597] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.288239 17105 consensus_peers.cc:597] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.284893 17031 consensus_peers.cc:597] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.289891 17031 consensus_peers.cc:597] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:57.290364 17031 consensus_peers.cc:597] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:57.294705 17856 raft_consensus.cc:359] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:57.295492 17856 raft_consensus.cc:740] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
W20260709 13:05:57.297960 17030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 64708.1us
I20260709 13:05:57.298749 17856 consensus_queue.cc:260] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:05:57.302320 17028 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 66710.2us
I20260709 13:05:57.304075 17856 ts_tablet_manager.cc:1434] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 13:05:57.305193 17856 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:57.311543 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 86281.9us
I20260709 13:05:57.329568 17856 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.330569 17856 tablet_bootstrap.cc:492] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.331218 17856 ts_tablet_manager.cc:1403] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.026s	user 0.018s	sys 0.003s
I20260709 13:05:57.333961 17856 raft_consensus.cc:359] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.336576 17856 raft_consensus.cc:740] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.339897 17856 consensus_queue.cc:260] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.344409 17856 ts_tablet_manager.cc:1434] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 13:05:57.345371 17856 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.387233 17856 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.388365 17856 tablet_bootstrap.cc:492] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.389129 17856 ts_tablet_manager.cc:1403] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.044s	user 0.010s	sys 0.012s
I20260709 13:05:57.392417 17856 raft_consensus.cc:359] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:57.393199 17856 raft_consensus.cc:740] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.393988 17856 consensus_queue.cc:260] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:57.396767 17856 ts_tablet_manager.cc:1434] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:05:57.397706 17856 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.437273 17856 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.438357 17856 tablet_bootstrap.cc:492] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.440361 17856 ts_tablet_manager.cc:1403] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.043s	user 0.018s	sys 0.009s
I20260709 13:05:57.443320 17856 raft_consensus.cc:359] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:57.444128 17856 raft_consensus.cc:740] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.445075 17856 consensus_queue.cc:260] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:57.447506 17856 ts_tablet_manager.cc:1434] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:05:57.448381 17856 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.484980 17856 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.486035 17856 tablet_bootstrap.cc:492] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.486806 17856 ts_tablet_manager.cc:1403] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.039s	user 0.015s	sys 0.010s
I20260709 13:05:57.491050 17856 raft_consensus.cc:359] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.491837 17856 raft_consensus.cc:740] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.492990 17856 consensus_queue.cc:260] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.497822 17856 ts_tablet_manager.cc:1434] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 13:05:57.498697 17856 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.554288 17856 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.555389 17856 tablet_bootstrap.cc:492] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.556102 17856 ts_tablet_manager.cc:1403] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.058s	user 0.025s	sys 0.024s
I20260709 13:05:57.558987 17856 raft_consensus.cc:359] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:57.559918 17856 raft_consensus.cc:740] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.560830 17856 consensus_queue.cc:260] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:57.563462 17856 ts_tablet_manager.cc:1434] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:05:57.564366 17856 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.590909 17856 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.591957 17856 tablet_bootstrap.cc:492] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.592664 17856 ts_tablet_manager.cc:1403] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.029s	user 0.017s	sys 0.004s
I20260709 13:05:57.595443 17856 raft_consensus.cc:359] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.596207 17856 raft_consensus.cc:740] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.597169 17856 consensus_queue.cc:260] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.599802 17856 ts_tablet_manager.cc:1434] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 13:05:57.600687 17856 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:57.640785 17031 consensus_peers.cc:597] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:57.643220 17856 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.644313 17856 tablet_bootstrap.cc:492] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.644982 17856 ts_tablet_manager.cc:1403] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.009s
I20260709 13:05:57.647960 17856 raft_consensus.cc:359] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.648723 17856 raft_consensus.cc:740] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.649642 17856 consensus_queue.cc:260] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.652288 17856 ts_tablet_manager.cc:1434] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 13:05:57.653203 17856 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.685652 17856 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.686745 17856 tablet_bootstrap.cc:492] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.687525 17856 ts_tablet_manager.cc:1403] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.035s	user 0.014s	sys 0.008s
I20260709 13:05:57.690627 17856 raft_consensus.cc:359] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.691413 17856 raft_consensus.cc:740] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.692409 17856 consensus_queue.cc:260] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.698993 17856 ts_tablet_manager.cc:1434] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:05:57.699910 17856 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:57.724996 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 114931us
I20260709 13:05:57.726526 17856 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.727620 17856 tablet_bootstrap.cc:492] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.728343 17856 ts_tablet_manager.cc:1403] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.029s	user 0.018s	sys 0.007s
I20260709 13:05:57.731093 17856 raft_consensus.cc:359] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.732128 17856 raft_consensus.cc:740] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.732937 17856 consensus_queue.cc:260] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.735459 17856 ts_tablet_manager.cc:1434] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:05:57.736296 17856 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.758518 17856 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.759625 17856 tablet_bootstrap.cc:492] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.761473 17856 ts_tablet_manager.cc:1403] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.025s	user 0.020s	sys 0.003s
I20260709 13:05:57.764225 17856 raft_consensus.cc:359] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.764986 17856 raft_consensus.cc:740] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.765836 17856 consensus_queue.cc:260] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:57.768822 17856 ts_tablet_manager.cc:1434] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:05:57.769783 17856 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:57.792997 17856 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.794121 17856 tablet_bootstrap.cc:492] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.794906 17856 ts_tablet_manager.cc:1403] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.025s	user 0.023s	sys 0.000s
I20260709 13:05:57.797714 17856 raft_consensus.cc:359] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.798485 17856 raft_consensus.cc:740] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.799479 17856 consensus_queue.cc:260] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.802022 17856 ts_tablet_manager.cc:1434] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:05:57.802914 17856 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:57.822676 17030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 71384.1us
I20260709 13:05:57.835428 17856 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:57.844384 17856 tablet_bootstrap.cc:492] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:57.845139 17856 ts_tablet_manager.cc:1403] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.042s	user 0.024s	sys 0.003s
I20260709 13:05:57.847870 17856 raft_consensus.cc:359] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.848618 17856 raft_consensus.cc:740] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:57.849326 17856 consensus_queue.cc:260] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:57.855486 17856 ts_tablet_manager.cc:1434] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 13:05:57.856508 17856 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:57.898137 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 55009.2us
W20260709 13:05:57.948047 17105 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 92322.9us
I20260709 13:05:58.002923 17856 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.005065 17856 tablet_bootstrap.cc:492] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.005937 17856 ts_tablet_manager.cc:1403] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.150s	user 0.018s	sys 0.054s
I20260709 13:05:58.009087 17856 raft_consensus.cc:359] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.010143 17856 raft_consensus.cc:740] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.011000 17856 consensus_queue.cc:260] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.013777 17856 ts_tablet_manager.cc:1434] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 13:05:58.014822 17856 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:58.014758 16957 consensus_peers.cc:597] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:58.020448 17031 consensus_peers.cc:597] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:58.078058 17031 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:36803, user_credentials={real_user=slave}} blocked reactor thread for 56837.6us
I20260709 13:05:58.090387 17856 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.091707 17856 tablet_bootstrap.cc:492] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
W20260709 13:05:58.112462 16957 consensus_peers.cc:597] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:58.113895 17856 ts_tablet_manager.cc:1403] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.099s	user 0.018s	sys 0.005s
I20260709 13:05:58.116987 17856 raft_consensus.cc:359] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.117691 17856 raft_consensus.cc:740] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
W20260709 13:05:58.120867 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 52232.3us
I20260709 13:05:58.120807 17856 consensus_queue.cc:260] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.124655 17856 ts_tablet_manager.cc:1434] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:05:58.125569 17856 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:58.136251 17105 consensus_peers.cc:597] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:58.152048 17105 consensus_peers.cc:597] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:58.149050 17031 consensus_peers.cc:597] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:58.156265 17031 consensus_peers.cc:597] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:58.166942 17031 consensus_peers.cc:597] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:58.176330 17858 raft_consensus.cc:493] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:58.178521 17858 raft_consensus.cc:515] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.181720 17858 leader_election.cc:290] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:58.202683 17858 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:58.203248 17858 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.212100 17858 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:58.240168 17856 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.266984 17856 tablet_bootstrap.cc:492] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.267627 17856 ts_tablet_manager.cc:1403] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.142s	user 0.022s	sys 0.018s
I20260709 13:05:58.273255 17952 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:58.273912 17952 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.276489 17952 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:58.340795 17856 raft_consensus.cc:359] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.341634 17856 raft_consensus.cc:740] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.347865 17856 consensus_queue.cc:260] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
W20260709 13:05:58.356588 16957 consensus_peers.cc:597] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:58.361545 17856 ts_tablet_manager.cc:1434] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.094s	user 0.004s	sys 0.007s
I20260709 13:05:58.369573 17856 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.375665 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:58.378940 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:58.398267 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:58.398793 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65c685d378244491a760363b4eec896e" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:58.402468 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:58.403851 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
W20260709 13:05:58.403407 17031 consensus_peers.cc:597] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:58.405680 17850 leader_election.cc:304] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:58.406293 17852 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:58.406635 17952 raft_consensus.cc:2749] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:58.407404 17952 raft_consensus.cc:2749] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:58.408434 17850 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:58.409904 17858 raft_consensus.cc:2749] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:05:58.412848 17031 consensus_peers.cc:597] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:58.423170 17856 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.424396 17856 tablet_bootstrap.cc:492] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.425163 17856 ts_tablet_manager.cc:1403] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.063s	user 0.012s	sys 0.004s
I20260709 13:05:58.428102 17856 raft_consensus.cc:359] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.428875 17856 raft_consensus.cc:740] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.433219 17856 consensus_queue.cc:260] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.436291 17856 ts_tablet_manager.cc:1434] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 13:05:58.437248 17856 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.478611  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:36803
I20260709 13:05:58.478880 17968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:36803 every 8 connection(s)
I20260709 13:05:58.482439 17858 raft_consensus.cc:493] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:58.483163 17858 raft_consensus.cc:515] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.487290 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:58.491359 17858 leader_election.cc:290] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:58.491600 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f5d356796924994baa41134536450d2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:58.494659 17852 leader_election.cc:304] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:05:58.499119 17858 raft_consensus.cc:2749] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:58.524724 17856 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.526114 17856 tablet_bootstrap.cc:492] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.527020 17856 ts_tablet_manager.cc:1403] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.090s	user 0.021s	sys 0.012s
I20260709 13:05:58.530522 17856 raft_consensus.cc:359] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.540678 17856 raft_consensus.cc:740] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.541949 17856 consensus_queue.cc:260] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.544836 17856 ts_tablet_manager.cc:1434] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.003s
I20260709 13:05:58.545850 17856 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.584491 17856 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.585783 17856 tablet_bootstrap.cc:492] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.586663 17856 ts_tablet_manager.cc:1403] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.041s	user 0.011s	sys 0.007s
I20260709 13:05:58.592829 17920 raft_consensus.cc:1240] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Rejecting Update request from peer c9521ca6efa94c3e9f074df69170cd5b for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:58.596560 17858 raft_consensus.cc:493] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:58.601191 17858 raft_consensus.cc:515] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.595924 17856 raft_consensus.cc:359] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.602559 17856 raft_consensus.cc:740] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.603497 17858 leader_election.cc:290] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:58.608428 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:58.603438 17856 consensus_queue.cc:260] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.611042 17856 ts_tablet_manager.cc:1434] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.001s
I20260709 13:05:58.612056 17856 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.614672 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:58.616298 17850 leader_election.cc:304] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:58.618072 17885 consensus_queue.cc:1059] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:58.618850 17858 raft_consensus.cc:2749] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:58.619191 17885 raft_consensus.cc:3055] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:58.619585 17885 raft_consensus.cc:740] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:58.620446 17885 consensus_queue.cc:260] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.621970 17885 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:58.627173 17921 raft_consensus.cc:1240] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Rejecting Update request from peer c9521ca6efa94c3e9f074df69170cd5b for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:58.629948 17858 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:58.630520 17858 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.632727 17858 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:05:58.637316 17864 consensus_queue.cc:1059] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:58.639283 17864 raft_consensus.cc:3055] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:58.639693 17864 raft_consensus.cc:740] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:58.640553 17864 consensus_queue.cc:260] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.656584 17864 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:58.683976 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:05:58.682802 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:58.690232 17856 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.690402 17858 raft_consensus.cc:493] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:58.691344 17856 tablet_bootstrap.cc:492] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.691030 17858 raft_consensus.cc:515] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.692173 17856 ts_tablet_manager.cc:1403] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.080s	user 0.008s	sys 0.010s
I20260709 13:05:58.693746 17858 leader_election.cc:290] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:58.694334 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:58.703125 17856 raft_consensus.cc:359] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.703980 17856 raft_consensus.cc:740] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.704842 17856 consensus_queue.cc:260] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.711807 17919 raft_consensus.cc:3060] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:58.712410 17850 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:58.713624 17952 raft_consensus.cc:2749] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:58.716194 17856 ts_tablet_manager.cc:1434] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.004s
I20260709 13:05:58.717134 17856 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.719198 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95585e8984cc45859857b2ff7777c409" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:58.727452 17850 leader_election.cc:304] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:58.728629 17858 raft_consensus.cc:2749] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:05:58.729987 17920 raft_consensus.cc:3060] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:58.735833 17921 raft_consensus.cc:3060] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:58.750339 16957 consensus_peers.cc:597] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:58.759932 17856 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.762524 17856 tablet_bootstrap.cc:492] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.763433 17856 ts_tablet_manager.cc:1403] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.047s	user 0.021s	sys 0.000s
I20260709 13:05:58.766590 17856 raft_consensus.cc:359] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.767402 17856 raft_consensus.cc:740] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.777478 17856 consensus_queue.cc:260] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.786896 17856 ts_tablet_manager.cc:1434] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 13:05:58.788033 17856 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.840116 17970 heartbeater.cc:344] Connected to a master server at 127.7.78.126:37799
I20260709 13:05:58.840481 17856 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.841099 17970 heartbeater.cc:461] Registering TS with master...
I20260709 13:05:58.842108 17856 tablet_bootstrap.cc:492] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.842741 17970 heartbeater.cc:507] Master 127.7.78.126:37799 requested a full tablet report, sending...
I20260709 13:05:58.843300 17856 ts_tablet_manager.cc:1403] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.056s	user 0.017s	sys 0.007s
I20260709 13:05:58.847431 17856 raft_consensus.cc:359] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.848412 17856 raft_consensus.cc:740] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.853799 17856 consensus_queue.cc:260] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:58.857355 17856 ts_tablet_manager.cc:1434] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.001s
I20260709 13:05:58.858659 17856 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.893198 17978 raft_consensus.cc:493] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:58.893944 17978 raft_consensus.cc:515] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:58.920760 17978 leader_election.cc:290] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:58.923028 17918 raft_consensus.cc:3060] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:58.957731 17985 log.cc:927] Time spent T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531: Append to log took a long time: real 0.058s	user 0.024s	sys 0.009s
I20260709 13:05:58.965361 17856 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:58.960651 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:58.966511 17856 tablet_bootstrap.cc:492] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:58.967244 17856 ts_tablet_manager.cc:1403] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.109s	user 0.020s	sys 0.001s
I20260709 13:05:58.970981 17856 raft_consensus.cc:359] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.971747 17856 raft_consensus.cc:740] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:58.972467 17856 consensus_queue.cc:260] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:58.975797 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46c332c6b3e148348daba6e06b3bc967" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:58.978538 17856 ts_tablet_manager.cc:1434] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 13:05:58.980062 17856 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:58.987437 16809 ts_manager.cc:194] Re-registered known tserver with Master: 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:05:58.987675 17850 leader_election.cc:304] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:58.990057 17921 raft_consensus.cc:3060] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:58.990485 17858 raft_consensus.cc:2749] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:05:58.997668 17920 raft_consensus.cc:3060] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:59.017659 17921 raft_consensus.cc:3060] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:59.056428 17856 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.057572 17856 tablet_bootstrap.cc:492] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.058225 17856 ts_tablet_manager.cc:1403] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.078s	user 0.007s	sys 0.014s
I20260709 13:05:59.060890 17856 raft_consensus.cc:359] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:59.061900 17856 raft_consensus.cc:740] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.062835 17856 consensus_queue.cc:260] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:59.065442 17917 raft_consensus.cc:3060] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:59.075538 17856 ts_tablet_manager.cc:1434] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:05:59.081588 17856 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.119285 17856 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.120318 17856 tablet_bootstrap.cc:492] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.120990 17856 ts_tablet_manager.cc:1403] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.040s	user 0.015s	sys 0.004s
I20260709 13:05:59.124150 17856 raft_consensus.cc:359] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.124922 17856 raft_consensus.cc:740] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.125628 17856 consensus_queue.cc:260] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.127156 17856 ts_tablet_manager.cc:1434] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 13:05:59.127938 17856 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.150517 16809 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51236
I20260709 13:05:59.154274 17856 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.155417 17856 tablet_bootstrap.cc:492] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.156113 17856 ts_tablet_manager.cc:1403] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.001s
I20260709 13:05:59.157522 17970 heartbeater.cc:499] Master 127.7.78.126:37799 was elected leader, sending a full tablet report...
I20260709 13:05:59.159587 17856 raft_consensus.cc:359] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.164669 17856 raft_consensus.cc:740] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.165616 17856 consensus_queue.cc:260] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.168129 17856 ts_tablet_manager.cc:1434] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 13:05:59.168989 17856 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:59.174551 17105 consensus_peers.cc:597] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:59.208282 17918 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:05:59.221802 17031 consensus_peers.cc:597] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:59.250272 17856 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.251473 17856 tablet_bootstrap.cc:492] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.252235 17856 ts_tablet_manager.cc:1403] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.083s	user 0.028s	sys 0.019s
I20260709 13:05:59.255302 17856 raft_consensus.cc:359] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:59.256057 17856 raft_consensus.cc:740] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.262960 17856 consensus_queue.cc:260] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:05:59.265592 17856 ts_tablet_manager.cc:1434] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.001s
I20260709 13:05:59.266453 17856 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.290149 17856 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 13:05:59.290762 17105 consensus_peers.cc:597] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:59.291277 17856 tablet_bootstrap.cc:492] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.292078 17856 ts_tablet_manager.cc:1403] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.012s
I20260709 13:05:59.297691 17856 raft_consensus.cc:359] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.298481 17856 raft_consensus.cc:740] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.299325 17856 consensus_queue.cc:260] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.301002 17856 ts_tablet_manager.cc:1434] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:05:59.301887 17856 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:59.329375 16957 consensus_peers.cc:597] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:59.339735 17856 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.340839 17856 tablet_bootstrap.cc:492] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.341673 17856 ts_tablet_manager.cc:1403] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.013s
I20260709 13:05:59.344520 17856 raft_consensus.cc:359] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.345242 17856 raft_consensus.cc:740] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.346027 17856 consensus_queue.cc:260] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.348392 17856 ts_tablet_manager.cc:1434] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.002s
I20260709 13:05:59.349267 17856 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
W20260709 13:05:59.381412 17031 consensus_peers.cc:597] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:59.448467 17856 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.449658 17856 tablet_bootstrap.cc:492] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.450402 17856 ts_tablet_manager.cc:1403] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.101s	user 0.039s	sys 0.048s
I20260709 13:05:59.453024 17856 raft_consensus.cc:359] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.453835 17856 raft_consensus.cc:740] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.454674 17856 consensus_queue.cc:260] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.457087 17856 ts_tablet_manager.cc:1434] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260709 13:05:59.461194 17856 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.529681 17856 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.530727 17856 tablet_bootstrap.cc:492] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.531415 17856 ts_tablet_manager.cc:1403] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.072s	user 0.023s	sys 0.002s
I20260709 13:05:59.538916 17856 raft_consensus.cc:359] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.540728 17856 raft_consensus.cc:740] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.541558 17856 consensus_queue.cc:260] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.544024 17856 ts_tablet_manager.cc:1434] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:05:59.544664 17918 raft_consensus.cc:1240] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Rejecting Update request from peer c9521ca6efa94c3e9f074df69170cd5b for earlier term 1. Current term is 2. Ops: []
I20260709 13:05:59.549520 17885 consensus_queue.cc:1059] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:05:59.559058 17856 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.572391 17976 raft_consensus.cc:3055] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:05:59.573014 17976 raft_consensus.cc:740] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:05:59.575479 17976 consensus_queue.cc:260] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.580360 17976 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:05:59.585187 17031 consensus_peers.cc:597] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:59.612607 17856 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.613755 17856 tablet_bootstrap.cc:492] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.614478 17856 ts_tablet_manager.cc:1403] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.056s	user 0.016s	sys 0.001s
I20260709 13:05:59.617197 17856 raft_consensus.cc:359] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.617903 17856 raft_consensus.cc:740] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.618582 17856 consensus_queue.cc:260] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.620082 17856 ts_tablet_manager.cc:1434] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20260709 13:05:59.620927 17856 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.649082 17952 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:59.649785 17952 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.658605 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:59.664270 17952 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:59.675263 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:59.676059 16986 raft_consensus.cc:2468] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:59.687758 17852 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:59.690032 17952 raft_consensus.cc:2804] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:59.690621 17952 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:59.691103 17952 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:59.717378 17952 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.720338 17952 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
W20260709 13:05:59.727614 17031 consensus_peers.cc:597] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:59.730345 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:59.731417 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:59.732933 16989 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:59.743149 16989 raft_consensus.cc:2468] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 3.
I20260709 13:05:59.744354 17852 leader_election.cc:304] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:05:59.746160 18007 raft_consensus.cc:2804] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:05:59.746716 18007 raft_consensus.cc:697] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:05:59.747615 18007 consensus_queue.cc:237] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.753618 17917 raft_consensus.cc:3060] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:59.782729 16814 catalog_manager.cc:5697] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 3, leader changed from c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:59.812832 17856 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.818310 17856 tablet_bootstrap.cc:492] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.819020 17856 ts_tablet_manager.cc:1403] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.198s	user 0.070s	sys 0.082s
I20260709 13:05:59.821719 17856 raft_consensus.cc:359] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.822404 17856 raft_consensus.cc:740] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.823108 17856 consensus_queue.cc:260] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.825461 17856 ts_tablet_manager.cc:1434] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:05:59.826620 17856 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.841991 17920 raft_consensus.cc:1240] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Rejecting Update request from peer 40c8156681dd4a7bb2d037afbeb01580 for earlier term 2. Current term is 3. Ops: []
I20260709 13:05:59.845925 18005 consensus_queue.cc:1059] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:05:59.846990 18005 raft_consensus.cc:3055] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:05:59.847358 18005 raft_consensus.cc:740] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:05:59.848177 18005 consensus_queue.cc:260] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.864869 18005 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:59.872179 18003 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:59.872802 18003 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.874444 17856 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:59.875020 18003 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:05:59.875499 17856 tablet_bootstrap.cc:492] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:05:59.876190 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:59.876886 16989 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:05:59.877079 17952 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:59.877607 17952 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.879712 17952 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:59.876223 17856 ts_tablet_manager.cc:1403] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.050s	user 0.032s	sys 0.011s
I20260709 13:05:59.879719 17030 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:05:59.881542 17930 raft_consensus.cc:2804] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:05:59.881920 17930 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:59.882372 17930 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:59.883415 17856 raft_consensus.cc:359] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.884274 17856 raft_consensus.cc:740] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:05:59.885111 17856 consensus_queue.cc:260] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:05:59.886818 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:05:59.887928 17856 ts_tablet_manager.cc:1434] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:05:59.888993 17856 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:05:59.889286 17917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:05:59.890218 17917 raft_consensus.cc:2393] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b in current term 2: Already voted for candidate 71a10d4adec64d58ba645d68425ed531 in this term.
I20260709 13:05:59.891490 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:59.892129 16989 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:59.893150 17852 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:05:59.895270 17952 raft_consensus.cc:2804] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:05:59.895668 17952 raft_consensus.cc:493] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:59.896031 17952 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:59.911880 17930 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.915582 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:59.918681 17063 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:05:59.922451 17920 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531"
I20260709 13:05:59.918705 18003 raft_consensus.cc:493] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:05:59.927006 18003 raft_consensus.cc:515] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.956149 17952 raft_consensus.cc:515] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:05:59.957540 17920 raft_consensus.cc:2368] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b for earlier term 2. Current term is 3.
I20260709 13:05:59.959368 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:05:59.960160 16986 raft_consensus.cc:2368] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b for earlier term 2. Current term is 3.
I20260709 13:05:59.961596 17030 leader_election.cc:400] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Vote denied by peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991) with higher term. Message: Invalid argument: T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b for earlier term 2. Current term is 3.
I20260709 13:05:59.961884 17030 leader_election.cc:403] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:05:59.962646 18002 raft_consensus.cc:3060] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:05:59.968390 17952 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:05:59.979632 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:05:59.990917 17918 raft_consensus.cc:3060] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:05:59.992563 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34c0447f54954c6b9ad3b7eb4f9402fa" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:05:59.993636 17063 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:05:59.994184 17031 leader_election.cc:400] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Vote denied by peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803) with higher term. Message: Invalid argument: T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b for earlier term 2. Current term is 3.
I20260709 13:05:59.994556 17031 leader_election.cc:403] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:05:59.998977 17920 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9193bb7f35422380ce6a3fb390eef9" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:00.008601 18003 leader_election.cc:290] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:00.009948 17930 leader_election.cc:290] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 election: Requested vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:00.013185 17063 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 3.
I20260709 13:06:00.020756 17884 raft_consensus.cc:2749] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803) with higher term. Message: Invalid argument: T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b for earlier term 2. Current term is 3.
I20260709 13:05:59.998085 16989 raft_consensus.cc:2468] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83f5ad22c2547b29c99ae897ea9b61b in term 2.
I20260709 13:06:00.025708 17850 leader_election.cc:304] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:00.028812 18002 raft_consensus.cc:2749] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991) with higher term. Message: Invalid argument: T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f83f5ad22c2547b29c99ae897ea9b61b for earlier term 2. Current term is 3.
I20260709 13:06:00.030344 17952 raft_consensus.cc:2804] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:00.032152 17952 raft_consensus.cc:697] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:00.033047 17952 consensus_queue.cc:237] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:00.083025 17856 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:00.084039 17856 tablet_bootstrap.cc:492] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:06:00.084704 17856 ts_tablet_manager.cc:1403] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.196s	user 0.076s	sys 0.076s
I20260709 13:06:00.087260 17856 raft_consensus.cc:359] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:00.089273 17856 raft_consensus.cc:740] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:06:00.090358 17856 consensus_queue.cc:260] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:00.092886 17856 ts_tablet_manager.cc:1434] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:06:00.094897 17856 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:06:00.116199 17856 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:00.117259 17856 tablet_bootstrap.cc:492] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:06:00.118251 17856 ts_tablet_manager.cc:1403] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.025s	user 0.017s	sys 0.002s
I20260709 13:06:00.121094 17856 raft_consensus.cc:359] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:00.121866 17856 raft_consensus.cc:740] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:06:00.122951 17856 consensus_queue.cc:260] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:00.125427 17856 ts_tablet_manager.cc:1434] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:06:00.126447 17856 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:06:00.148530 17856 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:00.150292 17856 tablet_bootstrap.cc:492] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:06:00.149595 16814 catalog_manager.cc:5697] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 3, leader changed from c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:00.151198 17856 ts_tablet_manager.cc:1403] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.025s	user 0.022s	sys 0.000s
I20260709 13:06:00.154206 17856 raft_consensus.cc:359] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:00.154953 17856 raft_consensus.cc:740] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:06:00.155678 17856 consensus_queue.cc:260] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:00.158130 17856 ts_tablet_manager.cc:1434] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:06:00.158987 17856 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Bootstrap starting.
I20260709 13:06:00.185705 17856 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:00.186879 17856 tablet_bootstrap.cc:492] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Bootstrap complete.
I20260709 13:06:00.187697 17856 ts_tablet_manager.cc:1403] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Time spent bootstrapping tablet: real 0.029s	user 0.018s	sys 0.009s
I20260709 13:06:00.191830 16986 raft_consensus.cc:1275] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:00.193717 17856 raft_consensus.cc:359] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:00.194662 17856 raft_consensus.cc:740] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Initialized, Role: FOLLOWER
I20260709 13:06:00.195492 17856 consensus_queue.cc:260] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:00.198477 17856 ts_tablet_manager.cc:1434] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:06:00.198599 17952 consensus_queue.cc:1048] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:00.222817 17063 raft_consensus.cc:1275] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:00.224599 17918 raft_consensus.cc:1240] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Rejecting Update request from peer f83f5ad22c2547b29c99ae897ea9b61b for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:00.226787 18003 consensus_queue.cc:1059] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:00.227232 18007 consensus_queue.cc:1048] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:00.227979 18003 raft_consensus.cc:3055] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:00.228415 18003 raft_consensus.cc:740] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:00.230034 18003 consensus_queue.cc:260] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:00.231403 18003 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:00.348070 17918 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:00.351573 18007 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:00.352263 18007 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:00.354612 18007 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:00.356681 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:00.357355 16989 raft_consensus.cc:2468] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:00.358520 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:00.361143 17852 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:00.368331 18007 raft_consensus.cc:2804] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:06:00.368775 18007 raft_consensus.cc:493] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:00.369201 18007 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:00.378625 18007 raft_consensus.cc:515] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:00.380828 18007 leader_election.cc:290] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:00.381464 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:00.382596 16989 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:00.385542 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2aedbc9c254811b0e3bb47679bb443" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:00.392889 16989 raft_consensus.cc:2468] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 3.
I20260709 13:06:00.394537 17852 leader_election.cc:304] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:00.395907 18007 raft_consensus.cc:2804] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:00.396463 18007 raft_consensus.cc:697] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:00.397429 18007 consensus_queue.cc:237] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:00.433346 16814 catalog_manager.cc:5697] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 3, leader changed from c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:00.546522 17063 raft_consensus.cc:1275] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:00.548017 18007 consensus_queue.cc:1048] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:00.562877 16989 raft_consensus.cc:3060] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:00.579331 16989 raft_consensus.cc:1275] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:00.580740 18007 consensus_queue.cc:1048] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:00.826494 17056 raft_consensus.cc:3060] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:06:00.838004 17056 raft_consensus.cc:1275] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:00.841224 18007 consensus_queue.cc:1048] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:00.856747 16986 raft_consensus.cc:1275] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:00.859001 18031 consensus_queue.cc:1048] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:00.899099 18031 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:00.899674 18031 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:00.902082 18031 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:00.904201 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:00.904943 17131 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 3.
I20260709 13:06:00.907852 17850 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:06:00.910948 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:00.914661 18031 raft_consensus.cc:2804] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:06:00.915279 18031 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:00.916108 18031 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:00.924064 18031 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:00.926862 18031 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:00.927075 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:00.927464 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:00.927801 17131 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:00.935825 17131 raft_consensus.cc:2468] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 4.
I20260709 13:06:00.937000 17850 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:00.937819 18031 raft_consensus.cc:2804] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:00.938360 18031 raft_consensus.cc:697] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 4 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:00.939296 18031 consensus_queue.cc:237] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:00.952237 16814 catalog_manager.cc:5697] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 2 to 4, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 4 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:01.280800 18031 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:01.281338 18031 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:01.283560 18031 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:01.284921 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:01.286339 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:01.286990 17056 raft_consensus.cc:2468] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:01.288148 17850 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:06:01.291289 18031 raft_consensus.cc:2804] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:06:01.291770 18031 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:01.292181 18031 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:01.292901 17879 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:06:01.293480 17879 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:01.296391 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:01.296542 17879 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:06:01.297132 17131 raft_consensus.cc:2368] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b for earlier term 3. Current term is 4.
I20260709 13:06:01.296826 17918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:01.298640 16954 leader_election.cc:400] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Vote denied by peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323) with higher term. Message: Invalid argument: T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b for earlier term 3. Current term is 4.
I20260709 13:06:01.299047 16954 leader_election.cc:403] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:06:01.299674 17976 raft_consensus.cc:3060] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:06:01.301761 18031 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:01.305130 18031 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:01.306671 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:01.307262 17056 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:01.308665 17976 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323) with higher term. Message: Invalid argument: T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b for earlier term 3. Current term is 4.
I20260709 13:06:01.310967 16987 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:01.317183 17056 raft_consensus.cc:2468] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 3.
I20260709 13:06:01.318588 17850 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:01.319540 18031 raft_consensus.cc:2804] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:01.320389 18031 raft_consensus.cc:697] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:01.321381 18031 consensus_queue.cc:237] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:01.336565 16814 catalog_manager.cc:5697] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 3, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:01.440896 17131 raft_consensus.cc:1275] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:06:01.442735 18031 consensus_queue.cc:1048] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:01.462373 16987 raft_consensus.cc:1275] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 4 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:06:01.466840 17952 consensus_queue.cc:1048] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:01.742677 17976 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:01.744000 17976 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:01.753775 17917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:01.757344 17976 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:01.776553 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:01.777601 17063 raft_consensus.cc:2368] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b for earlier term 2. Current term is 3.
I20260709 13:06:01.779449 16954 leader_election.cc:400] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Vote denied by peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727) with higher term. Message: Invalid argument: T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b for earlier term 2. Current term is 3.
I20260709 13:06:01.779845 16954 leader_election.cc:403] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:06:01.780438 17976 raft_consensus.cc:3060] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:06:01.790302 17976 raft_consensus.cc:2749] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727) with higher term. Message: Invalid argument: T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9521ca6efa94c3e9f074df69170cd5b for earlier term 2. Current term is 3.
I20260709 13:06:01.856369 16986 raft_consensus.cc:1275] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:01.857786 17952 consensus_queue.cc:1048] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:01.874639 17063 raft_consensus.cc:1275] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:01.876502 17952 consensus_queue.cc:1048] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:03.782641 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 46959.7us
W20260709 13:06:03.815250 17102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 44838.3us
W20260709 13:06:03.841913 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:37727, user_credentials={real_user=slave}} blocked reactor thread for 55953.3us
W20260709 13:06:04.243680 16845 debug-util.cc:398] Leaking SignalData structure 0x7b0800314900 after lost signal to thread 7484
W20260709 13:06:04.244930 16845 debug-util.cc:398] Leaking SignalData structure 0x7b08003ba240 after lost signal to thread 16848
W20260709 13:06:04.246459 16845 debug-util.cc:398] Leaking SignalData structure 0x7b080000e340 after lost signal to thread 17015
W20260709 13:06:04.247555 16845 debug-util.cc:398] Leaking SignalData structure 0x7b0800109320 after lost signal to thread 17089
W20260709 13:06:04.248508 16845 debug-util.cc:398] Leaking SignalData structure 0x7b08001d1480 after lost signal to thread 17163
W20260709 13:06:04.249655 16845 debug-util.cc:398] Leaking SignalData structure 0x7b08001e20a0 after lost signal to thread 17968
W20260709 13:06:05.353123 17852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 63146.2us
I20260709 13:06:08.646865  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:08.709666 17059 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039"
dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36486
I20260709 13:06:08.710299 17059 raft_consensus.cc:606] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:08.725246 17138 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32"
dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:54004
I20260709 13:06:08.725881 17138 raft_consensus.cc:606] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:08.728490 17059 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "705e2e01faca442ea62979654d514c91"
dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36486
I20260709 13:06:08.729001 17059 raft_consensus.cc:606] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:08.731642 17059 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67"
dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36486
I20260709 13:06:08.732518 17059 raft_consensus.cc:606] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:08.735504 17138 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c"
dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
mode: GRACEFUL
new_leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
 from {username='slave'} at 127.0.0.1:54004
I20260709 13:06:08.736078 17138 raft_consensus.cc:606] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Received request to transfer leadership to TS c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:08.737591  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:06:08.739261  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:08.958876 18058 raft_consensus.cc:993] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:08.960477 18058 raft_consensus.cc:1081] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:08.961118 18085 raft_consensus.cc:993] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580: : Instructing follower c9521ca6efa94c3e9f074df69170cd5b to start an election
I20260709 13:06:08.961807 18058 raft_consensus.cc:1081] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Signalling peer c9521ca6efa94c3e9f074df69170cd5b to start an election
I20260709 13:06:08.962527 17919 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36744
I20260709 13:06:08.963187 17919 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:08.963613 17919 raft_consensus.cc:3060] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:08.963968 16986 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c"
dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
 from {username='slave'} at 127.0.0.1:33850
I20260709 13:06:08.964671 16986 raft_consensus.cc:493] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:08.965076 16986 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:08.973703 17919 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:08.973853 16986 raft_consensus.cc:515] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:08.975786 17919 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:08.976639 16986 leader_election.cc:290] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:08.976876 17138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:08.977773 17138 raft_consensus.cc:3055] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:08.977044 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6062bb4013504733ac85a70a2ae0ff32" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:08.978116 17138 raft_consensus.cc:740] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:06:08.978363 16989 raft_consensus.cc:3060] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:08.978026 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:08.979300 17059 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:08.979199 17138 consensus_queue.cc:260] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:08.981024 17138 raft_consensus.cc:3060] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:08.978605 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e73389947c4fad9319efe807e8016c" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:08.981695 17131 raft_consensus.cc:3055] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:08.981987 17131 raft_consensus.cc:740] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:06:08.982837 17131 consensus_queue.cc:260] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:08.984279 17131 raft_consensus.cc:3060] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:08.991001 17138 raft_consensus.cc:2468] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:08.992658 17850 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:06:08.993032 16989 raft_consensus.cc:2468] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:08.994210 18089 raft_consensus.cc:2804] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:08.994747 18089 raft_consensus.cc:697] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:08.995834 18089 consensus_queue.cc:237] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:08.997057 17131 raft_consensus.cc:2468] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 3.
I20260709 13:06:08.998894 16954 leader_election.cc:304] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:06:08.999807 18052 raft_consensus.cc:2804] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:09.000320 18052 raft_consensus.cc:697] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 3 LEADER]: Becoming Leader. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:06:09.001260 17059 raft_consensus.cc:2468] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9521ca6efa94c3e9f074df69170cd5b in term 3.
I20260709 13:06:09.001329 18052 consensus_queue.cc:237] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:09.019204 18079 raft_consensus.cc:993] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:09.019776 18079 raft_consensus.cc:1081] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:09.021893 17921 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36748
I20260709 13:06:09.021358 16814 catalog_manager.cc:5697] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b reported cstate change: term changed from 2 to 3, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66). New cstate: current_term: 3 leader_uuid: "c9521ca6efa94c3e9f074df69170cd5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:09.022536 17921 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:09.022924 17921 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:09.022987 16809 catalog_manager.cc:5697] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:09.032589 17921 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:09.035905 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:09.034976 17921 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:09.036718 17059 raft_consensus.cc:3055] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:09.037067 17059 raft_consensus.cc:740] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:09.038532 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e072a4fb708444f6ade1000459bc8c67" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:09.039153 17131 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:09.037952 17059 consensus_queue.cc:260] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:09.040951 17059 raft_consensus.cc:3060] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:09.047366 17131 raft_consensus.cc:2468] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 3.
I20260709 13:06:09.048914 17850 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:06:09.050015 18089 raft_consensus.cc:2804] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:09.050733 18089 raft_consensus.cc:697] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:09.051142 17059 raft_consensus.cc:2468] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 3.
I20260709 13:06:09.051852 18089 consensus_queue.cc:237] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:09.065547 16809 catalog_manager.cc:5697] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 2 to 3, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 3 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:09.165275 18079 raft_consensus.cc:993] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:09.166008 18082 raft_consensus.cc:1081] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:09.167874 17921 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "705e2e01faca442ea62979654d514c91"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36748
I20260709 13:06:09.168476 17921 raft_consensus.cc:493] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:09.168771 17921 raft_consensus.cc:3060] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:09.175487 17921 raft_consensus.cc:515] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:09.177296 17921 leader_election.cc:290] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:09.178314 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:09.178435 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "705e2e01faca442ea62979654d514c91" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:09.178980 17059 raft_consensus.cc:3055] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:09.179132 16989 raft_consensus.cc:3060] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:09.179342 17059 raft_consensus.cc:740] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:09.180451 17059 consensus_queue.cc:260] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:09.181962 17059 raft_consensus.cc:3060] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:09.187249 16989 raft_consensus.cc:2468] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:09.188271 17059 raft_consensus.cc:2468] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:09.188625 17852 leader_election.cc:304] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:06:09.189728 18089 raft_consensus.cc:2804] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:09.190408 18089 raft_consensus.cc:697] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:09.192281 18089 consensus_queue.cc:237] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:09.204520 16809 catalog_manager.cc:5697] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:09.291185 18079 raft_consensus.cc:993] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:09.291743 18082 raft_consensus.cc:1081] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:09.293552 17921 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36748
I20260709 13:06:09.294028 17921 raft_consensus.cc:493] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:09.294320 17921 raft_consensus.cc:3060] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:09.300416 17921 raft_consensus.cc:515] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:09.302021 17921 leader_election.cc:290] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:09.302847 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:09.303265 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17f3f3ca934e4998ab856ebde5eda039" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:09.303579 17059 raft_consensus.cc:3055] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:09.303925 17131 raft_consensus.cc:3060] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:09.303978 17059 raft_consensus.cc:740] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:09.305230 17059 consensus_queue.cc:260] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:09.306703 17059 raft_consensus.cc:3060] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:09.310765 17131 raft_consensus.cc:2468] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:09.313253 17059 raft_consensus.cc:2468] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:09.313273 17850 leader_election.cc:304] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:06:09.314410 18089 raft_consensus.cc:2804] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:09.314900 18089 raft_consensus.cc:697] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:09.315788 18089 consensus_queue.cc:237] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:09.324646 16809 catalog_manager.cc:5697] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:09.386201 17059 raft_consensus.cc:1275] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:09.388038 18052 consensus_queue.cc:1048] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:09.427048 17132 raft_consensus.cc:1275] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer c9521ca6efa94c3e9f074df69170cd5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:09.440896 18106 consensus_queue.cc:1048] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:06:09.510403 17131 raft_consensus.cc:1275] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:09.513193 18089 consensus_queue.cc:1048] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:09.526252 17059 raft_consensus.cc:1275] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:09.530959 18078 consensus_queue.cc:1048] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:09.559090 17131 raft_consensus.cc:1275] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:09.680653 18114 consensus_queue.cc:1048] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:09.708782 16986 raft_consensus.cc:1275] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:09.929189  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:06:10.458587 16956 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.005s	sys 0.010s
W20260709 13:06:10.459077 16956 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.005s	sys 0.010s
I20260709 13:06:10.465093 18107 raft_consensus.cc:493] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:06:10.465823 18107 raft_consensus.cc:515] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:10.467967 18107 leader_election.cc:290] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.471375 18115 consensus_queue.cc:1048] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260709 13:06:10.509006 17851 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.829s	user 0.006s	sys 0.009s
W20260709 13:06:10.510211 17851 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.830s	user 0.007s	sys 0.009s
W20260709 13:06:10.520604 17852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 49617.2us
I20260709 13:06:10.546047 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.625833 18122 raft_consensus.cc:493] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.626515 18121 raft_consensus.cc:493] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.626715 18122 raft_consensus.cc:515] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.627151 18121 raft_consensus.cc:515] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.629529 18121 leader_election.cc:290] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.630854 18122 leader_election.cc:290] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:06:10.631579 18121 raft_consensus.cc:493] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.632252 18121 raft_consensus.cc:515] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.634128 18117 raft_consensus.cc:493] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.634754 18117 raft_consensus.cc:515] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:10.636902 18117 leader_election.cc:290] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.637969 18121 leader_election.cc:290] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.642318 18124 raft_consensus.cc:493] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
W20260709 13:06:10.642663 16954 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.68:39323, user_credentials={real_user=slave}} blocked reactor thread for 148009us
I20260709 13:06:10.643083 18082 raft_consensus.cc:493] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:06:10.643716 18082 raft_consensus.cc:515] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:10.645476 18085 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.645776 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.646720 18082 leader_election.cc:290] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.647275 18082 raft_consensus.cc:493] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:06:10.647132 18085 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.647835 18082 raft_consensus.cc:515] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:10.650053 18085 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:06:10.646733 18117 raft_consensus.cc:493] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.650996 18126 raft_consensus.cc:493] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:06:10.643034 18124 raft_consensus.cc:515] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.655575 18082 leader_election.cc:290] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:06:10.641115 17852 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41991, user_credentials={real_user=slave}} blocked reactor thread for 92592.1us
I20260709 13:06:10.681496 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.679111 18126 raft_consensus.cc:515] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.684299 18052 raft_consensus.cc:493] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.681149 18117 raft_consensus.cc:515] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:10.689322 18117 leader_election.cc:290] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.659833 18078 raft_consensus.cc:493] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:06:10.690656 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.696604 18078 raft_consensus.cc:515] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.698823 18078 leader_election.cc:290] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.691138 16814 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51236 (request call id 44) took 1143 ms. Trace:
I20260709 13:06:10.700027 18078 raft_consensus.cc:493] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.693141 18115 raft_consensus.cc:493] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.700543 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.700778 18078 raft_consensus.cc:515] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.700937 18115 raft_consensus.cc:515] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.660107 18089 raft_consensus.cc:493] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:06:10.699863 16814 rpcz_store.cc:276] 0709 13:06:09.547832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:06:09.548662 (+   830us) service_pool.cc:224] Handling call
0709 13:06:10.691055 (+1142393us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34867,"reported_tablets":1}
I20260709 13:06:10.691774 16990 raft_consensus.cc:1275] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:10.693897 18124 leader_election.cc:290] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:06:10.702823 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.705687 18118 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:10.706251 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.706987 18052 raft_consensus.cc:515] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:10.709157 18126 leader_election.cc:290] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:06:10.711390 17059 raft_consensus.cc:1275] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:10.712764 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.714461 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.716104 17136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.716598 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.717116 16809 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58396 (request call id 144) took 1229 ms. Trace:
I20260709 13:06:10.735607 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffcadf1be2784b748c215778456c2384" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.721668 17138 raft_consensus.cc:1275] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:10.734918 16990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39702 (request call id 109) took 1011 ms. Trace:
I20260709 13:06:10.735616 16809 rpcz_store.cc:276] 0709 13:06:09.488015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:06:09.488185 (+   170us) service_pool.cc:224] Handling call
0709 13:06:10.717039 (+1228854us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260709 13:06:10.738312 16990 rpcz_store.cc:276] 0709 13:06:09.716870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:06:10.502072 (+785202us) service_pool.cc:224] Handling call
0709 13:06:10.660976 (+158904us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:06:10.724489 (+ 63513us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:06:10.727922 (+  3433us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:06:10.743177 17012 debug-util.cc:398] Leaking SignalData structure 0x7b08002dd180 after lost signal to thread 7484
W20260709 13:06:10.745776 17012 debug-util.cc:398] Leaking SignalData structure 0x7b080032a6a0 after lost signal to thread 16848
W20260709 13:06:10.747236 17012 debug-util.cc:398] Leaking SignalData structure 0x7b080016e720 after lost signal to thread 17015
W20260709 13:06:10.748430 17012 debug-util.cc:398] Leaking SignalData structure 0x7b08000f6360 after lost signal to thread 17089
W20260709 13:06:10.749738 17012 debug-util.cc:398] Leaking SignalData structure 0x7b08001256e0 after lost signal to thread 17163
I20260709 13:06:10.752383 18124 raft_consensus.cc:493] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:10.753031 18124 raft_consensus.cc:515] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:10.729135 18089 raft_consensus.cc:515] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.756428 18089 leader_election.cc:290] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.729439 18118 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:10.757812 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c0204bc7fd34bcda060984d39b24e99" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.759764 18115 leader_election.cc:290] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.761499 18078 leader_election.cc:290] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
W20260709 13:06:10.767738 17012 debug-util.cc:398] Leaking SignalData structure 0x7b0800269960 after lost signal to thread 17968
W20260709 13:06:10.768218 17012 debug-util.cc:398] Leaking SignalData structure 0x7b0800309c20 after lost signal to thread 17970
I20260709 13:06:10.770382 18134 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f83f5ad22c2547b29c99ae897ea9b61b)
I20260709 13:06:10.771147 18134 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.772264 16988 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea9f0699e38347ebaa746bfc5f3f1789" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:10.773752 18126 raft_consensus.cc:493] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:06:10.774360 18126 raft_consensus.cc:515] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:10.783923 18126 leader_election.cc:290] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.785482 18134 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.786722 18134 raft_consensus.cc:493] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:06:10.787478 18134 raft_consensus.cc:515] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:10.789623 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db95bd0ca80745f09d5104034b88d18c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:10.790642 18134 leader_election.cc:290] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:10.791775 18134 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 40c8156681dd4a7bb2d037afbeb01580)
I20260709 13:06:10.792361 18134 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:10.800575 17915 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:10.805790 18124 leader_election.cc:290] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:10.806706 18052 leader_election.cc:290] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:10.809203 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9915d72aefda43ec9b9f65e31cf0b9ea" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.811283 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.800076 17917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b961524f09a42878311bbfebdbf23de" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:10.802695 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.819149 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.809232 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c38d4f3da4f646a285221f607421196a" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.834093 17912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:10.792923 18118 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:06:10.803283 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c63f7d1e40424ea2d7f83316591ac2" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.845083 16954 leader_election.cc:304] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.846274 17913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2517faf22ec44930814616834f5c6b19" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:10.850208 17030 leader_election.cc:304] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:10.861773 18128 raft_consensus.cc:2749] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:10.862062 17030 leader_election.cc:304] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:10.862890 18117 raft_consensus.cc:2749] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:10.852523 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c4f7877dcc446269524421e26325689" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:10.853384 18115 consensus_queue.cc:1048] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:10.854311 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:10.868609 18128 raft_consensus.cc:2749] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:10.869015 17105 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.870064 18122 raft_consensus.cc:2749] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:10.870697 17030 leader_election.cc:304] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:10.872299 18124 raft_consensus.cc:2749] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:10.875072 18115 raft_consensus.cc:493] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9521ca6efa94c3e9f074df69170cd5b)
I20260709 13:06:10.875830 18115 raft_consensus.cc:515] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:10.878202 18115 leader_election.cc:290] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:10.878599 17105 leader_election.cc:304] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.885934 18122 raft_consensus.cc:2749] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:10.888011 18134 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:10.898607 17918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:10.898746 17061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.891386 16954 leader_election.cc:304] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.901141 18117 raft_consensus.cc:2749] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:10.901906 17917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4a541f57ac040ee9de0e73269605ab5" candidate_uuid: "c9521ca6efa94c3e9f074df69170cd5b" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:10.902832 16954 leader_election.cc:304] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.909080 18137 consensus_queue.cc:1048] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:10.916981 17060 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.920522 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64212ad563c34c83812b266155d7086a" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.923516 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:10.923969 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ee862553fe342db878b5b9582fb233c" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.915206 18117 raft_consensus.cc:2749] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:10.925581 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.926180 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b" is_pre_election: true
I20260709 13:06:10.928399 17059 raft_consensus.cc:1275] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:10.937424 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.942162 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.950450 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8c8eadfc13d42fc83c0bbdfbc0a98dd" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.952068 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.952621 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a02d063e934712ace429f988380d0f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.953022 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5db30b4886e54685a69ed279bec9df94" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:10.957135 17031 leader_election.cc:304] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:10.962924 17850 leader_election.cc:304] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.967343 17850 leader_election.cc:304] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.972687 17031 leader_election.cc:304] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:10.966044 18134 consensus_queue.cc:1048] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:06:10.980051 18130 raft_consensus.cc:2749] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:10.980388 18115 raft_consensus.cc:2749] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:10.987584 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" is_pre_election: true
I20260709 13:06:10.989660 17917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aa0046c7a4f453c8c3f27e71c92fbe2" candidate_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "71a10d4adec64d58ba645d68425ed531" is_pre_election: true
I20260709 13:06:11.000104 16957 leader_election.cc:304] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:10.969735 18137 raft_consensus.cc:2749] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:10.976477 18124 raft_consensus.cc:2749] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:11.006218 16957 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9521ca6efa94c3e9f074df69170cd5b; no voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:11.008428 18052 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:06:11.011360 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4b312c29c472f8c19d32d3029ee76" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:11.008885 18118 raft_consensus.cc:2749] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:11.029091 17850 leader_election.cc:304] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:11.030429 17850 leader_election.cc:304] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:11.033217 17850 leader_election.cc:304] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:11.004226 18078 consensus_queue.cc:1048] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 13:06:11.036548 18089 raft_consensus.cc:2749] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:11.044247 18136 raft_consensus.cc:2749] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:11.044502 18133 raft_consensus.cc:2749] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:11.046412 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40c8156681dd4a7bb2d037afbeb01580" is_pre_election: true
I20260709 13:06:11.071874 17850 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:11.076349 18121 raft_consensus.cc:2749] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:11.097779 17850 leader_election.cc:304] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:11.099581 18121 raft_consensus.cc:2749] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:11.101365 17850 leader_election.cc:304] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:11.104724  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:11.119135 18121 raft_consensus.cc:2749] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:11.123808 17850 leader_election.cc:304] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:11.132601 17850 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531; no voters: 40c8156681dd4a7bb2d037afbeb01580, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:11.138168 18134 raft_consensus.cc:2749] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:11.137964 18121 raft_consensus.cc:2749] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:11.170028 17031 leader_election.cc:304] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83f5ad22c2547b29c99ae897ea9b61b; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:11.198015 18148 raft_consensus.cc:2749] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:11.276854 17060 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5"
dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36486
I20260709 13:06:11.278123 17060 raft_consensus.cc:606] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
W20260709 13:06:11.295925 18151 log.cc:927] Time spent T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 13:06:11.298309 17065 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50"
dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36486
I20260709 13:06:11.299139 17065 raft_consensus.cc:606] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:11.345513 16986 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6"
dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:43536
I20260709 13:06:11.346108 16986 raft_consensus.cc:606] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:11.349323 17059 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3"
dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
mode: GRACEFUL
new_leader_uuid: "40c8156681dd4a7bb2d037afbeb01580"
 from {username='slave'} at 127.0.0.1:36486
I20260709 13:06:11.350060 17059 raft_consensus.cc:606] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Received request to transfer leadership to TS 40c8156681dd4a7bb2d037afbeb01580
I20260709 13:06:11.364706 17065 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f190fb92000c417a92080b56288b075f"
dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36486
I20260709 13:06:11.365473 17065 raft_consensus.cc:606] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:11.367985  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:06:11.370045  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:11.478564 18118 raft_consensus.cc:993] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.479365 18084 raft_consensus.cc:1081] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.481065 17918 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36746
I20260709 13:06:11.481690 17918 raft_consensus.cc:493] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:11.482060 17918 raft_consensus.cc:3060] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.504927 17918 raft_consensus.cc:515] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:11.514937 17918 leader_election.cc:290] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
I20260709 13:06:11.516323 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:11.516647 17134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8c8535d6c24400ab8efb3d995b908e6" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:11.517051 16986 raft_consensus.cc:3055] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:11.517385 16986 raft_consensus.cc:740] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9521ca6efa94c3e9f074df69170cd5b, State: Running, Role: LEADER
I20260709 13:06:11.518522 16986 consensus_queue.cc:260] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:11.519821 16986 raft_consensus.cc:3060] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.531026 16986 raft_consensus.cc:2468] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:11.517462 17134 raft_consensus.cc:3060] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.532212 17852 leader_election.cc:304] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:06:11.533147 18133 raft_consensus.cc:2804] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:11.533711 18133 raft_consensus.cc:697] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:11.534651 18133 consensus_queue.cc:237] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:11.549716 17134 raft_consensus.cc:2468] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:11.591861 18131 raft_consensus.cc:993] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.593279 18068 raft_consensus.cc:1081] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.592844 16810 catalog_manager.cc:5697] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.595892 18131 raft_consensus.cc:993] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.596385 18079 raft_consensus.cc:1081] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.599139 17912 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36748
I20260709 13:06:11.599567 17921 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f190fb92000c417a92080b56288b075f"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36748
I20260709 13:06:11.600353 17912 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:11.600350 17921 raft_consensus.cc:493] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:11.601306 17921 raft_consensus.cc:3060] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.601336 17912 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:11.612192 17912 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:11.621537 17912 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:11.622102 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:11.622763 17059 raft_consensus.cc:3055] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:11.623063 17059 raft_consensus.cc:740] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:11.623822 17059 consensus_queue.cc:260] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:11.624842 17059 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:11.621376 17921 raft_consensus.cc:515] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:11.687566 17059 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 4.
I20260709 13:06:11.683589 17135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61b68459f0de48d98ad40888859c37f5" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:11.689121 17135 raft_consensus.cc:3060] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:11.694326 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:11.694977 16989 raft_consensus.cc:3060] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.696491 17921 leader_election.cc:290] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:11.704567 16989 raft_consensus.cc:2468] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:11.708456 17135 raft_consensus.cc:2468] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 4.
I20260709 13:06:11.711793 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f190fb92000c417a92080b56288b075f" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:11.718089 18079 raft_consensus.cc:993] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: : Instructing follower 40c8156681dd4a7bb2d037afbeb01580 to start an election
I20260709 13:06:11.719179 17065 raft_consensus.cc:3055] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:11.719597 18079 raft_consensus.cc:1081] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Signalling peer 40c8156681dd4a7bb2d037afbeb01580 to start an election
I20260709 13:06:11.719650 17065 raft_consensus.cc:740] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:11.720613 17065 consensus_queue.cc:260] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:11.721817 17131 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3"
dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
 from {username='slave'} at 127.0.0.1:58026
I20260709 13:06:11.721903 17065 raft_consensus.cc:3060] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.722259 17131 raft_consensus.cc:493] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:11.722527 17131 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:11.727133 16986 raft_consensus.cc:1240] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Rejecting Update request from peer f83f5ad22c2547b29c99ae897ea9b61b for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:11.728622 17065 raft_consensus.cc:2468] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:11.719172 17850 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:06:11.733108 18133 raft_consensus.cc:2804] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:11.734071 18133 raft_consensus.cc:697] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 4 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:11.716672 17852 leader_election.cc:304] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:06:11.728490 17131 raft_consensus.cc:515] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:11.736487 18133 consensus_queue.cc:237] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:11.742076 17131 leader_election.cc:290] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:11.745294 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:11.745934 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df062cbe424f401ea234f20cdb26eff3" candidate_uuid: "40c8156681dd4a7bb2d037afbeb01580" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:11.745970 17065 raft_consensus.cc:3055] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:11.746989 17065 raft_consensus.cc:740] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:11.749333 16989 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:11.751312 17065 consensus_queue.cc:260] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:11.748886 18089 raft_consensus.cc:2804] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:11.760879 17065 raft_consensus.cc:3060] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:11.787719 18128 raft_consensus.cc:993] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.788354 18102 raft_consensus.cc:1081] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:11.781777 18089 raft_consensus.cc:697] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:11.805157 17917 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36748
I20260709 13:06:11.806030 17917 raft_consensus.cc:493] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:11.806505 17917 raft_consensus.cc:3060] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.809001 16989 raft_consensus.cc:2468] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 3.
I20260709 13:06:11.825922 18089 consensus_queue.cc:237] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:11.826696 17065 raft_consensus.cc:2468] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c8156681dd4a7bb2d037afbeb01580 in term 3.
I20260709 13:06:11.827620 17104 leader_election.cc:304] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, c9521ca6efa94c3e9f074df69170cd5b; no voters: 
I20260709 13:06:11.828853 18122 raft_consensus.cc:2804] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:11.830365 18122 raft_consensus.cc:697] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 LEADER]: Becoming Leader. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:06:11.831485 18122 consensus_queue.cc:237] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } }
I20260709 13:06:11.832170 17917 raft_consensus.cc:515] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:11.834146 17917 leader_election.cc:290] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991), f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727)
I20260709 13:06:11.842453 16989 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9521ca6efa94c3e9f074df69170cd5b"
I20260709 13:06:11.845772 16989 raft_consensus.cc:3060] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.851060 17065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "696628411ba54b68a0ae0c4b77350e50" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:11.852051 17065 raft_consensus.cc:3055] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:11.852458 17065 raft_consensus.cc:740] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f83f5ad22c2547b29c99ae897ea9b61b, State: Running, Role: LEADER
I20260709 13:06:11.859133 17065 consensus_queue.cc:260] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:11.861737 17065 raft_consensus.cc:3060] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:11.861166 16810 catalog_manager.cc:5697] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 3 to 4, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 4 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:11.868891 16814 catalog_manager.cc:5697] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 reported cstate change: term changed from 2 to 3, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68). New cstate: current_term: 3 leader_uuid: "40c8156681dd4a7bb2d037afbeb01580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:11.868990 16810 catalog_manager.cc:5697] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.874949 17065 raft_consensus.cc:2468] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:11.876359 16989 raft_consensus.cc:2468] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:11.877611 17850 leader_election.cc:304] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b; no voters: 
I20260709 13:06:11.878399 18133 raft_consensus.cc:2804] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:11.878890 18133 raft_consensus.cc:697] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:11.882599 18133 consensus_queue.cc:237] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } }
I20260709 13:06:11.902343 16810 catalog_manager.cc:5697] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.979952 16989 raft_consensus.cc:1275] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:11.981578 18133 consensus_queue.cc:1048] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:11.996388 17131 raft_consensus.cc:1275] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:11.998685 18133 consensus_queue.cc:1048] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:12.192667 17135 raft_consensus.cc:1275] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:06:12.193949 18089 consensus_queue.cc:1048] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:12.211627 17065 raft_consensus.cc:1275] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 4 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:06:12.213598 18089 consensus_queue.cc:1048] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:12.295631 16986 raft_consensus.cc:1275] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:12.297122 18089 consensus_queue.cc:1048] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:12.310632 17065 raft_consensus.cc:1275] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:12.315766 18163 consensus_queue.cc:1048] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:12.316319 16989 raft_consensus.cc:1275] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:12.318912 18122 consensus_queue.cc:1048] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:12.322930 17059 raft_consensus.cc:1275] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:12.328008 18089 consensus_queue.cc:1048] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:12.355439 16986 raft_consensus.cc:1275] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:12.370419 18089 consensus_queue.cc:1048] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:12.371119  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:12.379706 17059 raft_consensus.cc:1275] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Refusing update from remote peer 40c8156681dd4a7bb2d037afbeb01580: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:12.382300 18122 consensus_queue.cc:1048] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:12.439718  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:12.478112 17131 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb"
dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
mode: GRACEFUL
new_leader_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:54004
I20260709 13:06:12.478785 17131 raft_consensus.cc:606] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Received request to transfer leadership to TS 71a10d4adec64d58ba645d68425ed531
I20260709 13:06:12.480218  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 13:06:12.481968  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:12.918768 18185 raft_consensus.cc:993] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580: : Instructing follower 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:12.919360 18122 raft_consensus.cc:1081] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Signalling peer 71a10d4adec64d58ba645d68425ed531 to start an election
I20260709 13:06:12.920825 17912 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb"
dest_uuid: "71a10d4adec64d58ba645d68425ed531"
 from {username='slave'} at 127.0.0.1:36744
I20260709 13:06:12.921473 17912 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:12.921752 17912 raft_consensus.cc:3060] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:12.930184 17912 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:12.932523 17912 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Requested vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68:39323)
I20260709 13:06:12.934206 17131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "40c8156681dd4a7bb2d037afbeb01580"
I20260709 13:06:12.934873 17131 raft_consensus.cc:3055] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:12.935158 17131 raft_consensus.cc:740] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40c8156681dd4a7bb2d037afbeb01580, State: Running, Role: LEADER
I20260709 13:06:12.935905 17131 consensus_queue.cc:260] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:12.937199 17131 raft_consensus.cc:3060] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:12.941792 17059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5afc790981c04080acad7b7cd96544eb" candidate_uuid: "71a10d4adec64d58ba645d68425ed531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f83f5ad22c2547b29c99ae897ea9b61b"
I20260709 13:06:12.942389 17059 raft_consensus.cc:3060] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:12.943924 17131 raft_consensus.cc:2468] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:12.945188 17850 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580, 71a10d4adec64d58ba645d68425ed531; no voters: 
I20260709 13:06:12.945993 18163 raft_consensus.cc:2804] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:12.946471 18163 raft_consensus.cc:697] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Becoming Leader. State: Replica: 71a10d4adec64d58ba645d68425ed531, State: Running, Role: LEADER
I20260709 13:06:12.947602 18163 consensus_queue.cc:237] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:12.949288 17059 raft_consensus.cc:2468] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a10d4adec64d58ba645d68425ed531 in term 2.
I20260709 13:06:12.961159 16811 catalog_manager.cc:5697] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 reported cstate change: term changed from 1 to 2, leader changed from 40c8156681dd4a7bb2d037afbeb01580 (127.7.78.68) to 71a10d4adec64d58ba645d68425ed531 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "71a10d4adec64d58ba645d68425ed531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:13.461345 17131 raft_consensus.cc:1275] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:13.462693 18078 consensus_queue.cc:1048] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:13.483220  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:13.490427 17059 raft_consensus.cc:1275] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Refusing update from remote peer 71a10d4adec64d58ba645d68425ed531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:13.494210 18078 consensus_queue.cc:1048] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:13.541354  7481 tablet_server.cc:179] TabletServer@127.7.78.65:36803 shutting down...
W20260709 13:06:13.551323 17105 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 190 similar messages]
W20260709 13:06:13.588105 17105 consensus_peers.cc:597] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.588987 17105 consensus_peers.cc:597] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.591414 17031 consensus_peers.cc:597] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.593182 16957 consensus_peers.cc:597] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.599001 17031 consensus_peers.cc:597] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.600126 16957 consensus_peers.cc:597] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.611814 17105 consensus_peers.cc:597] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.615839  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:06:13.616665 16957 consensus_peers.cc:597] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.617748  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.620837  7481 raft_consensus.cc:2243] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.621549  7481 raft_consensus.cc:2272] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.624905  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.625859  7481 raft_consensus.cc:2243] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.626811  7481 raft_consensus.cc:2272] T e072a4fb708444f6ade1000459bc8c67 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.630843  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.632002  7481 raft_consensus.cc:2243] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:13.632156 17031 consensus_peers.cc:597] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.632682  7481 raft_consensus.cc:2272] T 9fcfa9f9a5224b69b785589ce292ca3e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.636472  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.637212  7481 raft_consensus.cc:2243] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.637873  7481 raft_consensus.cc:2272] T c91680aaf7f642fb8b9697eaa08d67ac P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.640646  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.641386  7481 raft_consensus.cc:2243] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.641855  7481 raft_consensus.cc:2272] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.644603  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.645503  7481 raft_consensus.cc:2243] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.646530  7481 raft_consensus.cc:2272] T e4a541f57ac040ee9de0e73269605ab5 P 71a10d4adec64d58ba645d68425ed531 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.649446  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.650130  7481 raft_consensus.cc:2243] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.651023  7481 raft_consensus.cc:2272] T 6062bb4013504733ac85a70a2ae0ff32 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.653487  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.654316  7481 raft_consensus.cc:2243] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.655395  7481 raft_consensus.cc:2272] T 61b68459f0de48d98ad40888859c37f5 P 71a10d4adec64d58ba645d68425ed531 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.658105  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.658932  7481 raft_consensus.cc:2243] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.659554  7481 raft_consensus.cc:2272] T 64212ad563c34c83812b266155d7086a P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.662935  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.663605  7481 raft_consensus.cc:2243] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.664127  7481 raft_consensus.cc:2272] T b78448a359344202ad036ddd40ccee93 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.666421  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.667280  7481 raft_consensus.cc:2243] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.668022  7481 raft_consensus.cc:2272] T d582455c79c94516b94ee97bd42c5078 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.670585  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.671501  7481 raft_consensus.cc:2243] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.672487  7481 raft_consensus.cc:2272] T 5afc790981c04080acad7b7cd96544eb P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.676327  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.677206  7481 raft_consensus.cc:2243] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.677855  7481 raft_consensus.cc:2272] T 73e77860bb944a5e97539267e7d0f9ba P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.680763  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.681787  7481 raft_consensus.cc:2243] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.682363  7481 raft_consensus.cc:2272] T cd439fd5f4d4479f9d2c2003abed1c4c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.685586  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.686504  7481 raft_consensus.cc:2243] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.687745  7481 raft_consensus.cc:2272] T 4d9193bb7f35422380ce6a3fb390eef9 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:13.689332 17031 consensus_peers.cc:597] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.690963  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.692094  7481 raft_consensus.cc:2243] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.692801  7481 raft_consensus.cc:2272] T af54823c7c644c92b460fde468cdfaa0 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.696066  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.696779  7481 raft_consensus.cc:2243] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.697727  7481 raft_consensus.cc:2272] T 696628411ba54b68a0ae0c4b77350e50 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.700342  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.701190  7481 raft_consensus.cc:2243] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.702100  7481 raft_consensus.cc:2272] T 17f3f3ca934e4998ab856ebde5eda039 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.705615  7481 tablet_replica.cc:333] stopping tablet replica
W20260709 13:06:13.705751 17031 consensus_peers.cc:597] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.706663  7481 raft_consensus.cc:2243] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.707355  7481 raft_consensus.cc:2272] T a3fbc4f00185436cbce1d8e24e6d1e3a P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.710569  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.711303  7481 raft_consensus.cc:2243] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.712245  7481 raft_consensus.cc:2272] T d8c8535d6c24400ab8efb3d995b908e6 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.715358  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.716301  7481 raft_consensus.cc:2243] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.716823  7481 raft_consensus.cc:2272] T 6e5613a2ba864f379a89f716babe1879 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.720487  7481 tablet_replica.cc:333] stopping tablet replica
W20260709 13:06:13.720922 17105 consensus_peers.cc:597] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.721563  7481 raft_consensus.cc:2243] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:13.722221 16957 consensus_peers.cc:597] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.722306  7481 raft_consensus.cc:2272] T 9b961524f09a42878311bbfebdbf23de P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.726555  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.727528  7481 raft_consensus.cc:2243] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Raft consensus shutting down.
W20260709 13:06:13.727716 17031 consensus_peers.cc:597] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.728637  7481 raft_consensus.cc:2272] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.731971  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.732841  7481 raft_consensus.cc:2243] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.733366  7481 raft_consensus.cc:2272] T 3f5d356796924994baa41134536450d2 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:13.734761 17031 consensus_peers.cc:597] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.736372  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.737179  7481 raft_consensus.cc:2243] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.738063  7481 raft_consensus.cc:2272] T 34c0447f54954c6b9ad3b7eb4f9402fa P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.740608  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.741343  7481 raft_consensus.cc:2243] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.741998  7481 raft_consensus.cc:2272] T 7e5eff39a6814f6e8a37a48064b3dd2e P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.744895  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.745739  7481 raft_consensus.cc:2243] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.746697  7481 raft_consensus.cc:2272] T f190fb92000c417a92080b56288b075f P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.749583  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.750309  7481 raft_consensus.cc:2243] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.750896  7481 raft_consensus.cc:2272] T 277a6b1c93854935be35e234b98330d2 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.754076  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.755084  7481 raft_consensus.cc:2243] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.755684  7481 raft_consensus.cc:2272] T 0f90b95672134fac819741b525d2931b P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.759063  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.759946  7481 raft_consensus.cc:2243] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.760576  7481 raft_consensus.cc:2272] T 5db30b4886e54685a69ed279bec9df94 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:13.762928 17031 consensus_peers.cc:597] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.764171  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.765051  7481 raft_consensus.cc:2243] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.765771  7481 raft_consensus.cc:2272] T 46c332c6b3e148348daba6e06b3bc967 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.768996  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.769896  7481 raft_consensus.cc:2243] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:13.770006 17105 consensus_peers.cc:597] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.770581  7481 raft_consensus.cc:2272] T 6c0204bc7fd34bcda060984d39b24e99 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.773092  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.773823  7481 raft_consensus.cc:2243] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.774312  7481 raft_consensus.cc:2272] T 71be301ca5434f9b8880e7a03ef1ebca P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.776597  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.777232  7481 raft_consensus.cc:2243] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.777762  7481 raft_consensus.cc:2272] T 65c685d378244491a760363b4eec896e P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.780083  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.780758  7481 raft_consensus.cc:2243] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.781829  7481 raft_consensus.cc:2272] T 9c2aedbc9c254811b0e3bb47679bb443 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.785079  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.786034  7481 raft_consensus.cc:2243] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.786688  7481 raft_consensus.cc:2272] T 2517faf22ec44930814616834f5c6b19 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.788866  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.789573  7481 raft_consensus.cc:2243] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.790055  7481 raft_consensus.cc:2272] T ffcadf1be2784b748c215778456c2384 P 71a10d4adec64d58ba645d68425ed531 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.792527  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.793123  7481 raft_consensus.cc:2243] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.793625  7481 raft_consensus.cc:2272] T d4a02d063e934712ace429f988380d0f P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.796281  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.797079  7481 raft_consensus.cc:2243] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.797684  7481 raft_consensus.cc:2272] T 95585e8984cc45859857b2ff7777c409 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:13.799003 16957 consensus_peers.cc:597] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.800284  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.801019  7481 raft_consensus.cc:2243] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.801631  7481 raft_consensus.cc:2272] T b2784691feee4fdc945ee8ea311ea7da P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.804073  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.804850  7481 raft_consensus.cc:2243] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:13.805939  7481 raft_consensus.cc:2272] T 705e2e01faca442ea62979654d514c91 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.809381  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.810278  7481 raft_consensus.cc:2243] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.810820  7481 raft_consensus.cc:2272] T 7ee862553fe342db878b5b9582fb233c P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.814572  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.815490  7481 raft_consensus.cc:2243] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.816126  7481 raft_consensus.cc:2272] T 5a3da49eeaa7457e8f3b1848d46bcea5 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:13.819628  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.820324  7481 raft_consensus.cc:2243] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.820932  7481 raft_consensus.cc:2272] T aa31fcff86ca4af6b0cb7c4be6736c57 P 71a10d4adec64d58ba645d68425ed531 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:13.822815 17105 consensus_peers.cc:597] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.824401  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:13.825161  7481 raft_consensus.cc:2243] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:13.825702  7481 raft_consensus.cc:2272] T bda4b312c29c472f8c19d32d3029ee76 P 71a10d4adec64d58ba645d68425ed531 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:13.832718 17105 consensus_peers.cc:597] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.844483 17105 consensus_peers.cc:597] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.851612 17031 consensus_peers.cc:597] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.851800 17105 consensus_peers.cc:597] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.898451 17105 consensus_peers.cc:597] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:13.917747  7481 tablet_server.cc:196] TabletServer@127.7.78.65:36803 shutdown complete.
W20260709 13:06:13.951982 17031 consensus_peers.cc:597] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:13.959046 16957 consensus_peers.cc:597] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.006911 17105 consensus_peers.cc:597] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.011283 16957 consensus_peers.cc:597] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.024993  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
I20260709 13:06:14.087446  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:14.088531  7481 tablet_replica.cc:333] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.089653  7481 raft_consensus.cc:2243] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.090730  7481 raft_consensus.cc:2272] T e5e73389947c4fad9319efe807e8016c P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.093274  7481 tablet_replica.cc:333] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.094172  7481 raft_consensus.cc:2243] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.094708  7481 raft_consensus.cc:2272] T 9fcfa9f9a5224b69b785589ce292ca3e P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.097267  7481 tablet_replica.cc:333] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.098294  7481 raft_consensus.cc:2243] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.098944  7481 raft_consensus.cc:2272] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.103389  7481 tablet_replica.cc:333] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.104696  7481 raft_consensus.cc:2243] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.105599  7481 raft_consensus.cc:2272] T 34c0447f54954c6b9ad3b7eb4f9402fa P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.109110  7481 tablet_replica.cc:333] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.110183  7481 raft_consensus.cc:2243] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.111464  7481 raft_consensus.cc:2272] T 3f5d356796924994baa41134536450d2 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.114626 16957 consensus_peers.cc:597] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.115228  7481 tablet_replica.cc:333] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.116655  7481 raft_consensus.cc:2243] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.117414  7481 raft_consensus.cc:2272] T df062cbe424f401ea234f20cdb26eff3 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.119763 17031 consensus_peers.cc:597] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.120854  7481 tablet_replica.cc:333] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.121811  7481 raft_consensus.cc:2243] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.122396  7481 raft_consensus.cc:2272] T e4a541f57ac040ee9de0e73269605ab5 P c9521ca6efa94c3e9f074df69170cd5b [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.124142 17030 consensus_peers.cc:597] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.125051  7481 tablet_replica.cc:333] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.125887  7481 raft_consensus.cc:2243] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.126468  7481 raft_consensus.cc:2272] T d4a02d063e934712ace429f988380d0f P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.129964  7481 tablet_replica.cc:333] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.130955  7481 raft_consensus.cc:2243] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
W20260709 13:06:14.131368 17104 consensus_peers.cc:597] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.132155  7481 raft_consensus.cc:2272] T ea9f0699e38347ebaa746bfc5f3f1789 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.136498  7481 tablet_replica.cc:333] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.137610  7481 raft_consensus.cc:2243] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.138980  7481 raft_consensus.cc:2272] T 73e77860bb944a5e97539267e7d0f9ba P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.142378  7481 tablet_replica.cc:333] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.143425  7481 raft_consensus.cc:2243] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.144176  7481 raft_consensus.cc:2272] T f4cf831311924bb7b8afbc30043e1550 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.147179 16957 consensus_peers.cc:597] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b -> Peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Couldn't send request to peer 71a10d4adec64d58ba645d68425ed531. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.157004  7481 tablet_replica.cc:333] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.158215  7481 raft_consensus.cc:2243] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.159204  7481 raft_consensus.cc:2272] T 696628411ba54b68a0ae0c4b77350e50 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.163417  7481 tablet_replica.cc:333] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.164422  7481 raft_consensus.cc:2243] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.166007  7481 raft_consensus.cc:2272] T af54823c7c644c92b460fde468cdfaa0 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.166684 17030 consensus_peers.cc:597] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.169349  7481 tablet_replica.cc:333] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.170550  7481 raft_consensus.cc:2243] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.171314  7481 raft_consensus.cc:2272] T f1adb90cc0554b10b2f20940047d6fc2 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.175379  7481 tablet_replica.cc:333] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.176483  7481 raft_consensus.cc:2243] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.177208  7481 raft_consensus.cc:2272] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.175175 17030 consensus_peers.cc:597] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.180517  7481 tablet_replica.cc:333] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.181485  7481 raft_consensus.cc:2243] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.182124  7481 raft_consensus.cc:2272] T 6e5613a2ba864f379a89f716babe1879 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.185272  7481 tablet_replica.cc:333] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.186290  7481 raft_consensus.cc:2243] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.186919  7481 raft_consensus.cc:2272] T 9915d72aefda43ec9b9f65e31cf0b9ea P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.190044  7481 tablet_replica.cc:333] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.191272  7481 raft_consensus.cc:2243] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.192420  7481 raft_consensus.cc:2272] T 3f4a7e79870e405e9938b891f964fd1d P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.196071  7481 tablet_replica.cc:333] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.197958  7481 raft_consensus.cc:2243] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.199283  7481 raft_consensus.cc:2272] T 7e5eff39a6814f6e8a37a48064b3dd2e P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.203989  7481 tablet_replica.cc:333] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
W20260709 13:06:14.204401 17104 consensus_peers.cc:597] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.205005  7481 raft_consensus.cc:2243] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.205786  7481 raft_consensus.cc:2272] T f190fb92000c417a92080b56288b075f P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.209190  7481 tablet_replica.cc:333] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
W20260709 13:06:14.211185 17030 consensus_peers.cc:597] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.214015 17030 consensus_peers.cc:597] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.216420  7481 raft_consensus.cc:2243] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.217187  7481 raft_consensus.cc:2272] T d8c8535d6c24400ab8efb3d995b908e6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.220947  7481 tablet_replica.cc:333] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.221895  7481 raft_consensus.cc:2243] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.222612  7481 raft_consensus.cc:2272] T a3fbc4f00185436cbce1d8e24e6d1e3a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.226099  7481 tablet_replica.cc:333] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.227188  7481 raft_consensus.cc:2243] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.227942  7481 raft_consensus.cc:2272] T cd439fd5f4d4479f9d2c2003abed1c4c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.231472  7481 tablet_replica.cc:333] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.232251  7481 raft_consensus.cc:2243] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.232877  7481 raft_consensus.cc:2272] T 4d9193bb7f35422380ce6a3fb390eef9 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.236128  7481 tablet_replica.cc:333] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
W20260709 13:06:14.237593 17104 consensus_peers.cc:597] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.238487  7481 raft_consensus.cc:2243] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.239636  7481 raft_consensus.cc:2272] T 95585e8984cc45859857b2ff7777c409 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.240725 17030 consensus_peers.cc:597] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.242372  7481 tablet_replica.cc:333] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.243144  7481 raft_consensus.cc:2243] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.243799  7481 raft_consensus.cc:2272] T c38d4f3da4f646a285221f607421196a P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.247071  7481 tablet_replica.cc:333] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.247936  7481 raft_consensus.cc:2243] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.249064  7481 raft_consensus.cc:2272] T c91680aaf7f642fb8b9697eaa08d67ac P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.251173 17030 consensus_peers.cc:597] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.252483  7481 tablet_replica.cc:333] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.253187  7481 raft_consensus.cc:2243] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.253712  7481 raft_consensus.cc:2272] T 0f90b95672134fac819741b525d2931b P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.255963  7481 tablet_replica.cc:333] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.256572  7481 raft_consensus.cc:2243] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.257592  7481 raft_consensus.cc:2272] T 46c332c6b3e148348daba6e06b3bc967 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.259716  7481 tablet_replica.cc:333] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.260236  7481 raft_consensus.cc:2243] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.260990  7481 raft_consensus.cc:2272] T 2c4f7877dcc446269524421e26325689 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.263186  7481 tablet_replica.cc:333] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.263829  7481 raft_consensus.cc:2243] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.264331  7481 raft_consensus.cc:2272] T aa31fcff86ca4af6b0cb7c4be6736c57 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.267138  7481 tablet_replica.cc:333] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.267812  7481 raft_consensus.cc:2243] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.268651  7481 raft_consensus.cc:2272] T bda4b312c29c472f8c19d32d3029ee76 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.270992  7481 tablet_replica.cc:333] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.271734  7481 raft_consensus.cc:2243] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.272256  7481 raft_consensus.cc:2272] T 3e32f5ae45c549ef88703600d6a9cde3 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.275753  7481 tablet_replica.cc:333] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.276571  7481 raft_consensus.cc:2243] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.277215  7481 raft_consensus.cc:2272] T b78448a359344202ad036ddd40ccee93 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.280133  7481 tablet_replica.cc:333] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.281004  7481 raft_consensus.cc:2243] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.282313  7481 raft_consensus.cc:2272] T 7b240339eb134181914367d7072b86b6 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.284960  7481 tablet_replica.cc:333] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.285828  7481 raft_consensus.cc:2243] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.286350  7481 raft_consensus.cc:2272] T d582455c79c94516b94ee97bd42c5078 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.288810  7481 tablet_replica.cc:333] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.289379  7481 raft_consensus.cc:2243] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.289887  7481 raft_consensus.cc:2272] T 11c63f7d1e40424ea2d7f83316591ac2 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.292351  7481 tablet_replica.cc:333] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.293123  7481 raft_consensus.cc:2243] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.293661  7481 raft_consensus.cc:2272] T 5a3da49eeaa7457e8f3b1848d46bcea5 P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.296657  7481 tablet_replica.cc:333] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.297497  7481 raft_consensus.cc:2243] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.298017  7481 raft_consensus.cc:2272] T 6062bb4013504733ac85a70a2ae0ff32 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.300002  7481 tablet_replica.cc:333] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.300683  7481 raft_consensus.cc:2243] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.301208  7481 raft_consensus.cc:2272] T 2521b7f56acb452087e4eccf58dd0f4a P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.303301  7481 tablet_replica.cc:333] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.303836  7481 raft_consensus.cc:2243] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.304735  7481 raft_consensus.cc:2272] T db95bd0ca80745f09d5104034b88d18c P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.306938  7481 tablet_replica.cc:333] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.307467  7481 raft_consensus.cc:2243] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.307958  7481 raft_consensus.cc:2272] T 9c2aedbc9c254811b0e3bb47679bb443 P c9521ca6efa94c3e9f074df69170cd5b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.309839  7481 tablet_replica.cc:333] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.310349  7481 raft_consensus.cc:2243] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.311167  7481 raft_consensus.cc:2272] T b2784691feee4fdc945ee8ea311ea7da P c9521ca6efa94c3e9f074df69170cd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.315791  7481 tablet_replica.cc:333] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b: stopping tablet replica
I20260709 13:06:14.316509  7481 raft_consensus.cc:2243] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.317075  7481 raft_consensus.cc:2272] T 705e2e01faca442ea62979654d514c91 P c9521ca6efa94c3e9f074df69170cd5b [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.359071 17104 consensus_peers.cc:597] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.368842 17104 consensus_peers.cc:597] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.400339 17104 consensus_peers.cc:597] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.411314 17030 consensus_peers.cc:597] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.411852 17030 consensus_peers.cc:597] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.435156 17104 consensus_peers.cc:597] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.463761  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
W20260709 13:06:14.493739 17104 consensus_peers.cc:597] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.550637 17104 consensus_peers.cc:597] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.559121  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
I20260709 13:06:14.586879 18206 raft_consensus.cc:493] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:14.588083 18206 raft_consensus.cc:515] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:14.591180 18206 leader_election.cc:290] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
W20260709 13:06:14.599844 17104 leader_election.cc:336] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111)
W20260709 13:06:14.603509 17105 leader_election.cc:336] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:06:14.604050 17105 leader_election.cc:304] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:14.604739 18206 raft_consensus.cc:2749] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:06:14.617985  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:14.619112  7481 tablet_replica.cc:333] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.620352  7481 raft_consensus.cc:2243] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.621192  7481 raft_consensus.cc:2272] T e5e73389947c4fad9319efe807e8016c P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.624370  7481 tablet_replica.cc:333] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.625345  7481 raft_consensus.cc:2243] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.626462  7481 raft_consensus.cc:2272] T 9b961524f09a42878311bbfebdbf23de P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.629654  7481 tablet_replica.cc:333] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.630604  7481 raft_consensus.cc:2243] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.631156  7481 raft_consensus.cc:2272] T 64212ad563c34c83812b266155d7086a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.634428  7481 tablet_replica.cc:333] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.635349  7481 raft_consensus.cc:2243] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:14.635708 17104 consensus_peers.cc:597] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 -> Peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Couldn't send request to peer c9521ca6efa94c3e9f074df69170cd5b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.635906  7481 raft_consensus.cc:2272] T e072a4fb708444f6ade1000459bc8c67 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.639565  7481 tablet_replica.cc:333] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.640236  7481 raft_consensus.cc:2243] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.640684  7481 raft_consensus.cc:2272] T df062cbe424f401ea234f20cdb26eff3 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.642828  7481 tablet_replica.cc:333] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.643425  7481 raft_consensus.cc:2243] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.643846  7481 raft_consensus.cc:2272] T 696628411ba54b68a0ae0c4b77350e50 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.646051  7481 tablet_replica.cc:333] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.646777  7481 raft_consensus.cc:2243] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.647289  7481 raft_consensus.cc:2272] T 73e77860bb944a5e97539267e7d0f9ba P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.649948  7481 tablet_replica.cc:333] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
W20260709 13:06:14.650496 17102 consensus_peers.cc:597] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.650583  7481 raft_consensus.cc:2243] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.651309  7481 raft_consensus.cc:2272] T 5aa0046c7a4f453c8c3f27e71c92fbe2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.653374  7481 tablet_replica.cc:333] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.654057  7481 raft_consensus.cc:2243] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.654577  7481 raft_consensus.cc:2272] T 34c0447f54954c6b9ad3b7eb4f9402fa P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.657121  7481 tablet_replica.cc:333] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.657877  7481 raft_consensus.cc:2243] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.658354  7481 raft_consensus.cc:2272] T ea9f0699e38347ebaa746bfc5f3f1789 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.658592 17102 consensus_peers.cc:597] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.660454  7481 tablet_replica.cc:333] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.661209  7481 raft_consensus.cc:2243] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.661705  7481 raft_consensus.cc:2272] T 17f3f3ca934e4998ab856ebde5eda039 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.663756  7481 tablet_replica.cc:333] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.664368  7481 raft_consensus.cc:2243] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.665181  7481 raft_consensus.cc:2272] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.667279  7481 tablet_replica.cc:333] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.667887  7481 raft_consensus.cc:2243] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.668578  7481 raft_consensus.cc:2272] T f4cf831311924bb7b8afbc30043e1550 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.670917  7481 tablet_replica.cc:333] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.671448  7481 raft_consensus.cc:2243] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.671886  7481 raft_consensus.cc:2272] T 2517faf22ec44930814616834f5c6b19 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.673982  7481 tablet_replica.cc:333] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.674499  7481 raft_consensus.cc:2243] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.674877  7481 raft_consensus.cc:2272] T db95bd0ca80745f09d5104034b88d18c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.676653  7481 tablet_replica.cc:333] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.677189  7481 raft_consensus.cc:2243] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.677644  7481 raft_consensus.cc:2272] T 9c2aedbc9c254811b0e3bb47679bb443 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.679551  7481 tablet_replica.cc:333] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.680068  7481 raft_consensus.cc:2243] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.680845  7481 raft_consensus.cc:2272] T ffcadf1be2784b748c215778456c2384 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.682742  7481 tablet_replica.cc:333] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.683251  7481 raft_consensus.cc:2243] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.683991  7481 raft_consensus.cc:2272] T f1adb90cc0554b10b2f20940047d6fc2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.686826  7481 tablet_replica.cc:333] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.687800  7481 raft_consensus.cc:2243] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.688730  7481 raft_consensus.cc:2272] T 2c6703b86b3e4b4fad39ae2660fe6ff8 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.691339  7481 tablet_replica.cc:333] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.692008  7481 raft_consensus.cc:2243] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.692565  7481 raft_consensus.cc:2272] T 95585e8984cc45859857b2ff7777c409 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.694687  7481 tablet_replica.cc:333] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.695322  7481 raft_consensus.cc:2243] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.696262  7481 raft_consensus.cc:2272] T c38d4f3da4f646a285221f607421196a P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.698951  7481 tablet_replica.cc:333] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.699504  7481 raft_consensus.cc:2243] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.699945  7481 raft_consensus.cc:2272] T 46c332c6b3e148348daba6e06b3bc967 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.702036  7481 tablet_replica.cc:333] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.702711  7481 raft_consensus.cc:2243] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.703145  7481 raft_consensus.cc:2272] T 2c4f7877dcc446269524421e26325689 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.705197  7481 tablet_replica.cc:333] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.705771  7481 raft_consensus.cc:2243] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.706940  7481 raft_consensus.cc:2272] T 5db30b4886e54685a69ed279bec9df94 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.709116  7481 tablet_replica.cc:333] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.709820  7481 raft_consensus.cc:2243] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.710294  7481 raft_consensus.cc:2272] T 4d9193bb7f35422380ce6a3fb390eef9 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.712556 17102 consensus_peers.cc:597] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.713086  7481 tablet_replica.cc:333] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.713836  7481 raft_consensus.cc:2243] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.714450  7481 raft_consensus.cc:2272] T 3f4a7e79870e405e9938b891f964fd1d P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.717103  7481 tablet_replica.cc:333] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.717809  7481 raft_consensus.cc:2243] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.717813 18206 raft_consensus.cc:493] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:14.718464 18206 raft_consensus.cc:515] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:14.719059  7481 raft_consensus.cc:2272] T 9915d72aefda43ec9b9f65e31cf0b9ea P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.720880 18206 leader_election.cc:290] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
I20260709 13:06:14.722857  7481 tablet_replica.cc:333] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
W20260709 13:06:14.723724 17102 leader_election.cc:336] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:06:14.723855  7481 raft_consensus.cc:2243] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.724918  7481 raft_consensus.cc:2272] T aa31fcff86ca4af6b0cb7c4be6736c57 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.726325 17105 leader_election.cc:336] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:06:14.726784 17105 leader_election.cc:304] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:14.727408 18206 raft_consensus.cc:2749] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:14.728430  7481 tablet_replica.cc:333] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.729604  7481 raft_consensus.cc:2243] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.730182  7481 raft_consensus.cc:2272] T 3e32f5ae45c549ef88703600d6a9cde3 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.733644  7481 tablet_replica.cc:333] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.734736  7481 raft_consensus.cc:2243] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.735374  7481 raft_consensus.cc:2272] T 65c685d378244491a760363b4eec896e P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.738251  7481 tablet_replica.cc:333] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.739013  7481 raft_consensus.cc:2243] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.739529  7481 raft_consensus.cc:2272] T 705e2e01faca442ea62979654d514c91 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.742233  7481 tablet_replica.cc:333] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
W20260709 13:06:14.742872 17102 consensus_peers.cc:597] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.743014  7481 raft_consensus.cc:2243] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.743654  7481 raft_consensus.cc:2272] T f190fb92000c417a92080b56288b075f P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:14.744738 17102 consensus_peers.cc:597] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:14.746161 17102 consensus_peers.cc:597] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.746476  7481 tablet_replica.cc:333] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.747366  7481 raft_consensus.cc:2243] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.747923  7481 raft_consensus.cc:2272] T 277a6b1c93854935be35e234b98330d2 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.750685  7481 tablet_replica.cc:333] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.751331  7481 raft_consensus.cc:2243] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.751754  7481 raft_consensus.cc:2272] T 61b68459f0de48d98ad40888859c37f5 P f83f5ad22c2547b29c99ae897ea9b61b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.753700  7481 tablet_replica.cc:333] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.754292  7481 raft_consensus.cc:2243] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.755059  7481 raft_consensus.cc:2272] T b78448a359344202ad036ddd40ccee93 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.757033  7481 tablet_replica.cc:333] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.757637  7481 raft_consensus.cc:2243] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.758087  7481 raft_consensus.cc:2272] T 7b240339eb134181914367d7072b86b6 P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.760006  7481 tablet_replica.cc:333] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.760529  7481 raft_consensus.cc:2243] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.761453  7481 raft_consensus.cc:2272] T d582455c79c94516b94ee97bd42c5078 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.763513  7481 tablet_replica.cc:333] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.764156  7481 raft_consensus.cc:2243] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.764609  7481 raft_consensus.cc:2272] T 5afc790981c04080acad7b7cd96544eb P f83f5ad22c2547b29c99ae897ea9b61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.766996  7481 tablet_replica.cc:333] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.767688  7481 raft_consensus.cc:2243] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
W20260709 13:06:14.768147 17102 consensus_peers.cc:597] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.768704  7481 raft_consensus.cc:2272] T 2521b7f56acb452087e4eccf58dd0f4a P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.771266  7481 tablet_replica.cc:333] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.772010  7481 raft_consensus.cc:2243] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.772817  7481 raft_consensus.cc:2272] T 6c0204bc7fd34bcda060984d39b24e99 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.774824  7481 tablet_replica.cc:333] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.775295  7481 raft_consensus.cc:2243] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.776012  7481 raft_consensus.cc:2272] T 71be301ca5434f9b8880e7a03ef1ebca P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.778592  7481 tablet_replica.cc:333] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.779286  7481 raft_consensus.cc:2243] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.779793  7481 raft_consensus.cc:2272] T 11c63f7d1e40424ea2d7f83316591ac2 P f83f5ad22c2547b29c99ae897ea9b61b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.782711  7481 tablet_replica.cc:333] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.783419  7481 raft_consensus.cc:2243] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:14.783999  7481 raft_consensus.cc:2272] T 7ee862553fe342db878b5b9582fb233c P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.786897  7481 tablet_replica.cc:333] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b: stopping tablet replica
I20260709 13:06:14.787609  7481 raft_consensus.cc:2243] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:14.788664  7481 raft_consensus.cc:2272] T 5a3da49eeaa7457e8f3b1848d46bcea5 P f83f5ad22c2547b29c99ae897ea9b61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:14.805248 18206 raft_consensus.cc:493] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:14.805939 18206 raft_consensus.cc:515] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } } peers { permanent_uuid: "c9521ca6efa94c3e9f074df69170cd5b" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41991 } }
I20260709 13:06:14.807833 18206 leader_election.cc:290] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803), c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991)
W20260709 13:06:14.812577 17105 leader_election.cc:336] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
W20260709 13:06:14.813180 17104 leader_election.cc:336] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c9521ca6efa94c3e9f074df69170cd5b (127.7.78.66:41991): Network error: Client connection negotiation failed: client connection to 127.7.78.66:41991: connect: Connection refused (error 111)
I20260709 13:06:14.814023 17104 leader_election.cc:304] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, c9521ca6efa94c3e9f074df69170cd5b
I20260709 13:06:14.815037 18206 raft_consensus.cc:2749] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:14.826704 18122 raft_consensus.cc:493] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:14.827332 18122 raft_consensus.cc:515] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:14.830364 18122 leader_election.cc:290] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:06:14.830428 17102 consensus_peers.cc:597] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.832578 18122 raft_consensus.cc:493] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:14.833113 18122 raft_consensus.cc:515] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
W20260709 13:06:14.835117 17105 leader_election.cc:336] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:06:14.835314 18122 leader_election.cc:290] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:06:14.836519 17102 leader_election.cc:336] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:06:14.837005 17102 leader_election.cc:304] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
W20260709 13:06:14.840025 17105 leader_election.cc:336] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:06:14.842631 18122 raft_consensus.cc:2749] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:06:14.851439 17102 leader_election.cc:336] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:06:14.851897 17102 leader_election.cc:304] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:14.852595 18122 raft_consensus.cc:2749] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:14.874531  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
W20260709 13:06:14.937222 17102 consensus_peers.cc:597] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 -> Peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Couldn't send request to peer f83f5ad22c2547b29c99ae897ea9b61b. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.67:37727: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:14.959699  7481 tablet_server.cc:179] TabletServer@127.7.78.68:0 shutting down...
I20260709 13:06:14.979188 18206 raft_consensus.cc:493] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71a10d4adec64d58ba645d68425ed531)
I20260709 13:06:14.979789 18206 raft_consensus.cc:515] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83f5ad22c2547b29c99ae897ea9b61b" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37727 } } peers { permanent_uuid: "40c8156681dd4a7bb2d037afbeb01580" member_type: VOTER last_known_addr { host: "127.7.78.68" port: 39323 } } peers { permanent_uuid: "71a10d4adec64d58ba645d68425ed531" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36803 } }
I20260709 13:06:14.986009 18206 leader_election.cc:290] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727), 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803)
W20260709 13:06:14.987954 17102 leader_election.cc:336] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f83f5ad22c2547b29c99ae897ea9b61b (127.7.78.67:37727): Network error: Client connection negotiation failed: client connection to 127.7.78.67:37727: connect: Connection refused (error 111)
W20260709 13:06:14.995428 17105 leader_election.cc:336] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71a10d4adec64d58ba645d68425ed531 (127.7.78.65:36803): Network error: Client connection negotiation failed: client connection to 127.7.78.65:36803: connect: Connection refused (error 111)
I20260709 13:06:14.995958 17105 leader_election.cc:304] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40c8156681dd4a7bb2d037afbeb01580; no voters: 71a10d4adec64d58ba645d68425ed531, f83f5ad22c2547b29c99ae897ea9b61b
I20260709 13:06:14.996697 18206 raft_consensus.cc:2749] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:15.016934  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:15.018126  7481 tablet_replica.cc:333] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.018958  7481 raft_consensus.cc:2243] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.019554  7481 raft_consensus.cc:2272] T 9b961524f09a42878311bbfebdbf23de P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.021906  7481 tablet_replica.cc:333] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.022487  7481 raft_consensus.cc:2243] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.023351  7481 raft_consensus.cc:2272] T 9fcfa9f9a5224b69b785589ce292ca3e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.025481  7481 tablet_replica.cc:333] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.026098  7481 raft_consensus.cc:2243] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.026577  7481 raft_consensus.cc:2272] T e5e73389947c4fad9319efe807e8016c P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.028563  7481 tablet_replica.cc:333] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.029186  7481 raft_consensus.cc:2243] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.030081  7481 raft_consensus.cc:2272] T df062cbe424f401ea234f20cdb26eff3 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.032132  7481 tablet_replica.cc:333] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.032788  7481 raft_consensus.cc:2243] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.033640  7481 raft_consensus.cc:2272] T 64212ad563c34c83812b266155d7086a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.036331  7481 tablet_replica.cc:333] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.037088  7481 raft_consensus.cc:2243] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.037989  7481 raft_consensus.cc:2272] T 11c63f7d1e40424ea2d7f83316591ac2 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.040346  7481 tablet_replica.cc:333] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.040974  7481 raft_consensus.cc:2243] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.041851  7481 raft_consensus.cc:2272] T 7ee862553fe342db878b5b9582fb233c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.043789  7481 tablet_replica.cc:333] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.044369  7481 raft_consensus.cc:2243] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.044823  7481 raft_consensus.cc:2272] T ea9f0699e38347ebaa746bfc5f3f1789 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.047600  7481 tablet_replica.cc:333] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.048527  7481 raft_consensus.cc:2243] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.049757  7481 raft_consensus.cc:2272] T 2517faf22ec44930814616834f5c6b19 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.052361  7481 tablet_replica.cc:333] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.053093  7481 raft_consensus.cc:2243] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.053668  7481 raft_consensus.cc:2272] T db95bd0ca80745f09d5104034b88d18c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.055769  7481 tablet_replica.cc:333] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.056336  7481 raft_consensus.cc:2243] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.056751  7481 raft_consensus.cc:2272] T e4a541f57ac040ee9de0e73269605ab5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.058911  7481 tablet_replica.cc:333] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.059479  7481 raft_consensus.cc:2243] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.059902  7481 raft_consensus.cc:2272] T 3f5d356796924994baa41134536450d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.062224  7481 tablet_replica.cc:333] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.062821  7481 raft_consensus.cc:2243] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.063247  7481 raft_consensus.cc:2272] T e072a4fb708444f6ade1000459bc8c67 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.065321  7481 tablet_replica.cc:333] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.066051  7481 raft_consensus.cc:2243] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.066524  7481 raft_consensus.cc:2272] T e8c8eadfc13d42fc83c0bbdfbc0a98dd P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.068921  7481 tablet_replica.cc:333] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.069602  7481 raft_consensus.cc:2243] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.070065  7481 raft_consensus.cc:2272] T f4cf831311924bb7b8afbc30043e1550 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.071962  7481 tablet_replica.cc:333] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.072726  7481 raft_consensus.cc:2243] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.073181  7481 raft_consensus.cc:2272] T ffcadf1be2784b748c215778456c2384 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.075335  7481 tablet_replica.cc:333] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.075934  7481 raft_consensus.cc:2243] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.076367  7481 raft_consensus.cc:2272] T b2784691feee4fdc945ee8ea311ea7da P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.078362  7481 tablet_replica.cc:333] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.078917  7481 raft_consensus.cc:2243] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.079334  7481 raft_consensus.cc:2272] T af54823c7c644c92b460fde468cdfaa0 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.081180  7481 tablet_replica.cc:333] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.081784  7481 raft_consensus.cc:2243] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.082257  7481 raft_consensus.cc:2272] T 6c0204bc7fd34bcda060984d39b24e99 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.084308  7481 tablet_replica.cc:333] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.084878  7481 raft_consensus.cc:2243] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.085363  7481 raft_consensus.cc:2272] T 71be301ca5434f9b8880e7a03ef1ebca P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.087206  7481 tablet_replica.cc:333] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.087787  7481 raft_consensus.cc:2243] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.088579  7481 raft_consensus.cc:2272] T f1adb90cc0554b10b2f20940047d6fc2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.090418  7481 tablet_replica.cc:333] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.090961  7481 raft_consensus.cc:2243] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.091392  7481 raft_consensus.cc:2272] T d8c8535d6c24400ab8efb3d995b908e6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.093380  7481 tablet_replica.cc:333] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.093943  7481 raft_consensus.cc:2243] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.094363  7481 raft_consensus.cc:2272] T 17f3f3ca934e4998ab856ebde5eda039 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.096163  7481 tablet_replica.cc:333] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.096701  7481 raft_consensus.cc:2243] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.097496  7481 raft_consensus.cc:2272] T a3fbc4f00185436cbce1d8e24e6d1e3a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.099354  7481 tablet_replica.cc:333] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.099892  7481 raft_consensus.cc:2243] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.100349  7481 raft_consensus.cc:2272] T 7e5eff39a6814f6e8a37a48064b3dd2e P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.102370  7481 tablet_replica.cc:333] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.103147  7481 raft_consensus.cc:2243] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.103686  7481 raft_consensus.cc:2272] T 2c4f7877dcc446269524421e26325689 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.105918  7481 tablet_replica.cc:333] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.106523  7481 raft_consensus.cc:2243] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.107364  7481 raft_consensus.cc:2272] T 0f90b95672134fac819741b525d2931b P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.109556  7481 tablet_replica.cc:333] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.110321  7481 raft_consensus.cc:2243] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.110987  7481 raft_consensus.cc:2272] T 5db30b4886e54685a69ed279bec9df94 P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.113819  7481 tablet_replica.cc:333] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.114513  7481 raft_consensus.cc:2243] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.115535  7481 raft_consensus.cc:2272] T 6e5613a2ba864f379a89f716babe1879 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.117663  7481 tablet_replica.cc:333] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.118362  7481 raft_consensus.cc:2243] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.118881  7481 raft_consensus.cc:2272] T 9915d72aefda43ec9b9f65e31cf0b9ea P 40c8156681dd4a7bb2d037afbeb01580 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.121106  7481 tablet_replica.cc:333] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.121807  7481 raft_consensus.cc:2243] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.122290  7481 raft_consensus.cc:2272] T 3f4a7e79870e405e9938b891f964fd1d P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.124344  7481 tablet_replica.cc:333] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.124944  7481 raft_consensus.cc:2243] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.125473  7481 raft_consensus.cc:2272] T 2521b7f56acb452087e4eccf58dd0f4a P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.127490  7481 tablet_replica.cc:333] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.128079  7481 raft_consensus.cc:2243] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.128995  7481 raft_consensus.cc:2272] T cd439fd5f4d4479f9d2c2003abed1c4c P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.132273  7481 tablet_replica.cc:333] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.133034  7481 raft_consensus.cc:2243] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.133615  7481 raft_consensus.cc:2272] T 61b68459f0de48d98ad40888859c37f5 P 40c8156681dd4a7bb2d037afbeb01580 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.136003  7481 tablet_replica.cc:333] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.136652  7481 raft_consensus.cc:2243] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.137111  7481 raft_consensus.cc:2272] T 6062bb4013504733ac85a70a2ae0ff32 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.139571  7481 tablet_replica.cc:333] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.140265  7481 raft_consensus.cc:2243] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.141456  7481 raft_consensus.cc:2272] T d4a02d063e934712ace429f988380d0f P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.144119  7481 tablet_replica.cc:333] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.144696  7481 raft_consensus.cc:2243] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.145134  7481 raft_consensus.cc:2272] T c38d4f3da4f646a285221f607421196a P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.147333  7481 tablet_replica.cc:333] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.147902  7481 raft_consensus.cc:2243] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.148330  7481 raft_consensus.cc:2272] T bda4b312c29c472f8c19d32d3029ee76 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.150365  7481 tablet_replica.cc:333] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.150902  7481 raft_consensus.cc:2243] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.151717  7481 raft_consensus.cc:2272] T 3e32f5ae45c549ef88703600d6a9cde3 P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.153891  7481 tablet_replica.cc:333] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.154537  7481 raft_consensus.cc:2243] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.155678  7481 raft_consensus.cc:2272] T 65c685d378244491a760363b4eec896e P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.158797  7481 tablet_replica.cc:333] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.159641  7481 raft_consensus.cc:2243] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.160295  7481 raft_consensus.cc:2272] T c91680aaf7f642fb8b9697eaa08d67ac P 40c8156681dd4a7bb2d037afbeb01580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.162499  7481 tablet_replica.cc:333] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.163121  7481 raft_consensus.cc:2243] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.164042  7481 raft_consensus.cc:2272] T 277a6b1c93854935be35e234b98330d2 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.166697  7481 tablet_replica.cc:333] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.167228  7481 raft_consensus.cc:2243] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.167721  7481 raft_consensus.cc:2272] T 7b240339eb134181914367d7072b86b6 P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.169939  7481 tablet_replica.cc:333] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580: stopping tablet replica
I20260709 13:06:15.170423  7481 raft_consensus.cc:2243] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:15.170882  7481 raft_consensus.cc:2272] T 5afc790981c04080acad7b7cd96544eb P 40c8156681dd4a7bb2d037afbeb01580 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.278106  7481 tablet_server.cc:196] TabletServer@127.7.78.68:0 shutdown complete.
I20260709 13:06:15.418771  7481 master.cc:562] Master@127.7.78.126:37799 shutting down...
I20260709 13:06:15.462777  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:15.463771  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:15.464151  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P cd12eb484b26430aa64e7db64cb43f07: stopping tablet replica
I20260709 13:06:15.511309  7481 master.cc:584] Master@127.7.78.126:37799 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (40912 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 13:06:15.618716  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:43571
I20260709 13:06:15.619895  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:15.627029 18216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:15.627156 18213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:15.627202 18214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:15.627724  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:15.629078  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:15.629295  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:15.629834  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602375629816 us; error 0 us; skew 500 ppm
I20260709 13:06:15.630376  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:15.633103  7481 webserver.cc:533] Webserver started at http://127.7.78.126:41761/ using document root <none> and password file <none>
I20260709 13:06:15.633675  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:15.633886  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:15.634159  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:15.635404  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "d0f6df7450ad403dbb99034ba2facc14"
format_stamp: "Formatted at 2026-07-09 13:06:15 on dist-test-slave-50cs"
I20260709 13:06:15.640100  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:06:15.644219 18221 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:15.645346  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:06:15.645701  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "d0f6df7450ad403dbb99034ba2facc14"
format_stamp: "Formatted at 2026-07-09 13:06:15 on dist-test-slave-50cs"
I20260709 13:06:15.645990  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:15.661166  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:15.663121  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:15.707233  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:43571
I20260709 13:06:15.707350 18272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:43571 every 8 connection(s)
I20260709 13:06:15.712365 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:15.726024 18273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14: Bootstrap starting.
I20260709 13:06:15.731971 18273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.736616 18273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14: No bootstrap required, opened a new log
I20260709 13:06:15.738902 18273 raft_consensus.cc:359] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f6df7450ad403dbb99034ba2facc14" member_type: VOTER }
I20260709 13:06:15.739334 18273 raft_consensus.cc:385] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.739619 18273 raft_consensus.cc:740] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f6df7450ad403dbb99034ba2facc14, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.740262 18273 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f6df7450ad403dbb99034ba2facc14" member_type: VOTER }
I20260709 13:06:15.740773 18273 raft_consensus.cc:399] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:15.740998 18273 raft_consensus.cc:493] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:15.741305 18273 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.746311 18273 raft_consensus.cc:515] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f6df7450ad403dbb99034ba2facc14" member_type: VOTER }
I20260709 13:06:15.746891 18273 leader_election.cc:304] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f6df7450ad403dbb99034ba2facc14; no voters: 
I20260709 13:06:15.748129 18273 leader_election.cc:290] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:15.748445 18276 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:15.750088 18276 raft_consensus.cc:697] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 1 LEADER]: Becoming Leader. State: Replica: d0f6df7450ad403dbb99034ba2facc14, State: Running, Role: LEADER
I20260709 13:06:15.750856 18276 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f6df7450ad403dbb99034ba2facc14" member_type: VOTER }
I20260709 13:06:15.751303 18273 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:15.759130 18278 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f6df7450ad403dbb99034ba2facc14. Latest consensus state: current_term: 1 leader_uuid: "d0f6df7450ad403dbb99034ba2facc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f6df7450ad403dbb99034ba2facc14" member_type: VOTER } }
I20260709 13:06:15.760142 18278 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:15.759583 18277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f6df7450ad403dbb99034ba2facc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f6df7450ad403dbb99034ba2facc14" member_type: VOTER } }
I20260709 13:06:15.760771 18277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:15.762512 18286 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:15.770349 18286 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:15.771385  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:15.780184 18286 catalog_manager.cc:1383] Generated new cluster ID: 7e7473615c1d49a8a1530a184619e6a1
I20260709 13:06:15.780504 18286 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:15.815968 18286 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:15.817385 18286 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:15.829380 18286 catalog_manager.cc:6070] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14: Generated new TSK 0
I20260709 13:06:15.830051 18286 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:15.838274  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:15.845337 18294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:15.846472 18295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:15.851696  7481 server_base.cc:1061] running on GCE node
W20260709 13:06:15.851994 18297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:15.853494  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:15.853771  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:15.853957  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602375853937 us; error 0 us; skew 500 ppm
I20260709 13:06:15.854656  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:15.857290  7481 webserver.cc:533] Webserver started at http://127.7.78.65:40309/ using document root <none> and password file <none>
I20260709 13:06:15.857910  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:15.858122  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:15.858400  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:15.859594  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "aa72f0c464794888866a56f660990608"
format_stamp: "Formatted at 2026-07-09 13:06:15 on dist-test-slave-50cs"
I20260709 13:06:15.864809  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:06:15.868625 18302 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:15.869618  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:06:15.869989  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "aa72f0c464794888866a56f660990608"
format_stamp: "Formatted at 2026-07-09 13:06:15 on dist-test-slave-50cs"
I20260709 13:06:15.870420  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:15.885251  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:15.886729  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:15.888512  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:15.891211  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:15.891489  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:15.891765  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:15.891993  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:15.937788  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:45805
I20260709 13:06:15.937944 18364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:45805 every 8 connection(s)
I20260709 13:06:15.943100  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:15.951632 18369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:15.952499 18370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:15.957839  7481 server_base.cc:1061] running on GCE node
W20260709 13:06:15.958163 18372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:15.960104  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:15.960419  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:15.960642  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602375960622 us; error 0 us; skew 500 ppm
I20260709 13:06:15.961499  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:15.962453 18365 heartbeater.cc:344] Connected to a master server at 127.7.78.126:43571
I20260709 13:06:15.962929 18365 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:15.963929 18365 heartbeater.cc:507] Master 127.7.78.126:43571 requested a full tablet report, sending...
I20260709 13:06:15.964807  7481 webserver.cc:533] Webserver started at http://127.7.78.66:45669/ using document root <none> and password file <none>
I20260709 13:06:15.965612  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:15.965831  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:15.966144  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:15.966926 18238 ts_manager.cc:194] Registered new tserver with Master: aa72f0c464794888866a56f660990608 (127.7.78.65:45805)
I20260709 13:06:15.967557  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "31250b946a464780846c6ea7b435e870"
format_stamp: "Formatted at 2026-07-09 13:06:15 on dist-test-slave-50cs"
I20260709 13:06:15.969030 18238 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38238
I20260709 13:06:15.973354  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20260709 13:06:15.977851 18377 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:15.978880  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:06:15.979260  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "31250b946a464780846c6ea7b435e870"
format_stamp: "Formatted at 2026-07-09 13:06:15 on dist-test-slave-50cs"
I20260709 13:06:15.979722  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:15.999375  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:16.000761  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:16.003005  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:16.005659  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:16.005940  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:16.006222  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:16.006429  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:16.054131  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:37781
I20260709 13:06:16.054241 18439 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:37781 every 8 connection(s)
I20260709 13:06:16.059063  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:16.066715 18443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:16.067764 18444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:16.074587 18440 heartbeater.cc:344] Connected to a master server at 127.7.78.126:43571
W20260709 13:06:16.074580 18447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:16.075157  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:16.075340 18440 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:16.076440  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:06:16.076462 18440 heartbeater.cc:507] Master 127.7.78.126:43571 requested a full tablet report, sending...
W20260709 13:06:16.076850  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:16.077083  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602376077064 us; error 0 us; skew 500 ppm
I20260709 13:06:16.077824  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:16.078948 18238 ts_manager.cc:194] Registered new tserver with Master: 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:16.080353 18238 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38250
I20260709 13:06:16.081120  7481 webserver.cc:533] Webserver started at http://127.7.78.67:41367/ using document root <none> and password file <none>
I20260709 13:06:16.081866  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:16.082129  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:16.082438  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:16.083694  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "ac771fb495d84241b8a5dc7056ff239f"
format_stamp: "Formatted at 2026-07-09 13:06:16 on dist-test-slave-50cs"
I20260709 13:06:16.088766  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:06:16.092684 18451 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:16.093639  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:06:16.093969  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "ac771fb495d84241b8a5dc7056ff239f"
format_stamp: "Formatted at 2026-07-09 13:06:16 on dist-test-slave-50cs"
I20260709 13:06:16.094275  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:16.109493  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:16.110843  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:16.112347  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:16.114754  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:16.114990  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:16.115247  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:16.115438  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:16.161859  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:37131
I20260709 13:06:16.161959 18513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:37131 every 8 connection(s)
I20260709 13:06:16.177558 18514 heartbeater.cc:344] Connected to a master server at 127.7.78.126:43571
I20260709 13:06:16.177996 18514 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:16.178861 18514 heartbeater.cc:507] Master 127.7.78.126:43571 requested a full tablet report, sending...
I20260709 13:06:16.181140 18238 ts_manager.cc:194] Registered new tserver with Master: ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:16.181604  7481 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016149362s
I20260709 13:06:16.182811 18238 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38254
I20260709 13:06:16.210554 18238 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38270:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:16.214174 18238 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:16.290638 18330 tablet_service.cc:1511] Processing CreateTablet for tablet 11892abfca5043928cb0890fde5e66b3 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:06:16.292335 18330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11892abfca5043928cb0890fde5e66b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.294577 18329 tablet_service.cc:1511] Processing CreateTablet for tablet 9db35f4181aa46adb426b1bae078a0df (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:06:16.296205 18329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db35f4181aa46adb426b1bae078a0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.298538 18328 tablet_service.cc:1511] Processing CreateTablet for tablet f2156261889b40898ea60bd3002d1ccf (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:06:16.300062 18328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2156261889b40898ea60bd3002d1ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.305984 18405 tablet_service.cc:1511] Processing CreateTablet for tablet 11892abfca5043928cb0890fde5e66b3 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:06:16.306622 18326 tablet_service.cc:1511] Processing CreateTablet for tablet 07bc6bf2f52c4c418867c6bbc31dbc6a (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:06:16.307544 18405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11892abfca5043928cb0890fde5e66b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.308022 18326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07bc6bf2f52c4c418867c6bbc31dbc6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.310369 18404 tablet_service.cc:1511] Processing CreateTablet for tablet 9db35f4181aa46adb426b1bae078a0df (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:06:16.310642 18325 tablet_service.cc:1511] Processing CreateTablet for tablet ea09119560864fdabe6de1bde69b8f51 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:06:16.311890 18404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db35f4181aa46adb426b1bae078a0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.304859 18327 tablet_service.cc:1511] Processing CreateTablet for tablet db070baeeb2b411fb19fed64cc792ab8 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:06:16.314602 18403 tablet_service.cc:1511] Processing CreateTablet for tablet f2156261889b40898ea60bd3002d1ccf (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:06:16.316141 18403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2156261889b40898ea60bd3002d1ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.318706 18402 tablet_service.cc:1511] Processing CreateTablet for tablet db070baeeb2b411fb19fed64cc792ab8 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:06:16.320165 18402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db070baeeb2b411fb19fed64cc792ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.322710 18401 tablet_service.cc:1511] Processing CreateTablet for tablet 07bc6bf2f52c4c418867c6bbc31dbc6a (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:06:16.324286 18401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07bc6bf2f52c4c418867c6bbc31dbc6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.326814 18400 tablet_service.cc:1511] Processing CreateTablet for tablet ea09119560864fdabe6de1bde69b8f51 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:06:16.328325 18400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea09119560864fdabe6de1bde69b8f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.330669 18324 tablet_service.cc:1511] Processing CreateTablet for tablet 9b6dee6d90d94db3bc0e31f7748252d0 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:06:16.311890 18325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea09119560864fdabe6de1bde69b8f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.335114 18397 tablet_service.cc:1511] Processing CreateTablet for tablet 087ed505256448bfafd264d7cb18396f (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:06:16.338348 18323 tablet_service.cc:1511] Processing CreateTablet for tablet 7d9d73a52af14ff69e7996ebcc71131d (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:06:16.335748 18399 tablet_service.cc:1511] Processing CreateTablet for tablet 9b6dee6d90d94db3bc0e31f7748252d0 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:06:16.340065 18396 tablet_service.cc:1511] Processing CreateTablet for tablet 104363652bb743148f6c64f81c33060e (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:06:16.341061 18322 tablet_service.cc:1511] Processing CreateTablet for tablet 087ed505256448bfafd264d7cb18396f (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:06:16.333256 18398 tablet_service.cc:1511] Processing CreateTablet for tablet 7d9d73a52af14ff69e7996ebcc71131d (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:06:16.338348 18321 tablet_service.cc:1511] Processing CreateTablet for tablet 104363652bb743148f6c64f81c33060e (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:06:16.348378 18397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 087ed505256448bfafd264d7cb18396f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.357865 18398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9d73a52af14ff69e7996ebcc71131d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.358361 18324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6dee6d90d94db3bc0e31f7748252d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.361904 18323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9d73a52af14ff69e7996ebcc71131d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.365840 18327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db070baeeb2b411fb19fed64cc792ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.373049 18322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 087ed505256448bfafd264d7cb18396f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.378221 18321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104363652bb743148f6c64f81c33060e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.381493 18396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104363652bb743148f6c64f81c33060e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.386027 18399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6dee6d90d94db3bc0e31f7748252d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.418706 18479 tablet_service.cc:1511] Processing CreateTablet for tablet 11892abfca5043928cb0890fde5e66b3 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:06:16.420284 18479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11892abfca5043928cb0890fde5e66b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.422672 18478 tablet_service.cc:1511] Processing CreateTablet for tablet 9db35f4181aa46adb426b1bae078a0df (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:06:16.424190 18478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db35f4181aa46adb426b1bae078a0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.429766 18477 tablet_service.cc:1511] Processing CreateTablet for tablet f2156261889b40898ea60bd3002d1ccf (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:06:16.431440 18477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2156261889b40898ea60bd3002d1ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.442237 18474 tablet_service.cc:1511] Processing CreateTablet for tablet ea09119560864fdabe6de1bde69b8f51 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:06:16.443722 18476 tablet_service.cc:1511] Processing CreateTablet for tablet db070baeeb2b411fb19fed64cc792ab8 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:06:16.445045 18471 tablet_service.cc:1511] Processing CreateTablet for tablet 087ed505256448bfafd264d7cb18396f (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:06:16.447494 18470 tablet_service.cc:1511] Processing CreateTablet for tablet 104363652bb743148f6c64f81c33060e (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:06:16.450282 18475 tablet_service.cc:1511] Processing CreateTablet for tablet 07bc6bf2f52c4c418867c6bbc31dbc6a (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:06:16.450275 18473 tablet_service.cc:1511] Processing CreateTablet for tablet 9b6dee6d90d94db3bc0e31f7748252d0 (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:06:16.452780 18529 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.443816 18474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea09119560864fdabe6de1bde69b8f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.443827 18472 tablet_service.cc:1511] Processing CreateTablet for tablet 7d9d73a52af14ff69e7996ebcc71131d (DEFAULT_TABLE table=test-workload [id=384c8ce31b5a46c48b8b65e6acc0d68d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:06:16.455116 18472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9d73a52af14ff69e7996ebcc71131d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.457799 18470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104363652bb743148f6c64f81c33060e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.458940 18471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 087ed505256448bfafd264d7cb18396f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.466665 18530 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.477655 18529 tablet_bootstrap.cc:654] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.483461 18473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6dee6d90d94db3bc0e31f7748252d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.484314 18476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db070baeeb2b411fb19fed64cc792ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.486994 18475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07bc6bf2f52c4c418867c6bbc31dbc6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:16.496933 18530 tablet_bootstrap.cc:654] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.527632 18533 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.537597 18533 tablet_bootstrap.cc:654] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.545171 18529 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:16.545948 18529 ts_tablet_manager.cc:1403] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.094s	user 0.014s	sys 0.005s
I20260709 13:06:16.550211 18530 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:16.550789 18530 ts_tablet_manager.cc:1403] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.085s	user 0.015s	sys 0.027s
I20260709 13:06:16.554840 18530 raft_consensus.cc:359] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.555603 18530 raft_consensus.cc:385] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.556438 18530 raft_consensus.cc:740] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.557384 18530 consensus_queue.cc:260] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.562067 18530 ts_tablet_manager.cc:1434] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 13:06:16.562970 18365 heartbeater.cc:499] Master 127.7.78.126:43571 was elected leader, sending a full tablet report...
I20260709 13:06:16.563287 18533 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:16.563905 18533 ts_tablet_manager.cc:1403] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.037s	user 0.009s	sys 0.005s
I20260709 13:06:16.565896 18529 raft_consensus.cc:359] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.570874 18529 raft_consensus.cc:385] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.571254 18529 raft_consensus.cc:740] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.566918 18533 raft_consensus.cc:359] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.572075 18529 consensus_queue.cc:260] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.572635 18533 raft_consensus.cc:385] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.573015 18533 raft_consensus.cc:740] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.567795 18530 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.578063 18533 consensus_queue.cc:260] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.585834 18440 heartbeater.cc:499] Master 127.7.78.126:43571 was elected leader, sending a full tablet report...
I20260709 13:06:16.586753 18514 heartbeater.cc:499] Master 127.7.78.126:43571 was elected leader, sending a full tablet report...
I20260709 13:06:16.587100 18530 tablet_bootstrap.cc:654] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.593299 18529 ts_tablet_manager.cc:1434] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.008s
I20260709 13:06:16.593570 18533 ts_tablet_manager.cc:1434] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.007s
I20260709 13:06:16.595052 18533 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.595350 18529 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.600419 18530 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:16.601046 18530 ts_tablet_manager.cc:1403] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.006s
I20260709 13:06:16.601804 18533 tablet_bootstrap.cc:654] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.604507 18530 raft_consensus.cc:359] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.605619 18530 raft_consensus.cc:385] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.605952 18530 raft_consensus.cc:740] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.606933 18530 consensus_queue.cc:260] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.607999 18533 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:16.608053 18529 tablet_bootstrap.cc:654] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.608881 18533 ts_tablet_manager.cc:1403] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.005s
I20260709 13:06:16.609910 18530 ts_tablet_manager.cc:1434] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 13:06:16.611119 18530 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.612417 18533 raft_consensus.cc:359] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.613349 18533 raft_consensus.cc:385] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.613754 18533 raft_consensus.cc:740] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.614895 18533 consensus_queue.cc:260] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.617641 18533 ts_tablet_manager.cc:1434] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:16.618815 18533 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.620990 18529 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:16.621610 18529 ts_tablet_manager.cc:1403] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.009s
I20260709 13:06:16.621626 18530 tablet_bootstrap.cc:654] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.625078 18529 raft_consensus.cc:359] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.626190 18529 raft_consensus.cc:385] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.626579 18533 tablet_bootstrap.cc:654] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.626583 18529 raft_consensus.cc:740] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.628312 18529 consensus_queue.cc:260] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.631646 18529 ts_tablet_manager.cc:1434] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:06:16.632658 18529 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.633807 18530 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:16.634403 18530 ts_tablet_manager.cc:1403] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.012s
I20260709 13:06:16.637769 18530 raft_consensus.cc:359] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.638527 18530 raft_consensus.cc:385] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.638841 18530 raft_consensus.cc:740] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.639655 18529 tablet_bootstrap.cc:654] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.639708 18530 consensus_queue.cc:260] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.642755 18533 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:16.643476 18533 ts_tablet_manager.cc:1403] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.013s
I20260709 13:06:16.643994 18530 ts_tablet_manager.cc:1434] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:06:16.645493 18530 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.653119 18533 raft_consensus.cc:359] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.654631 18533 raft_consensus.cc:385] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.655242 18533 raft_consensus.cc:740] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.656097 18533 consensus_queue.cc:260] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.658540 18530 tablet_bootstrap.cc:654] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.658655 18533 ts_tablet_manager.cc:1434] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.003s
I20260709 13:06:16.659974 18533 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.660938 18529 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:16.661553 18529 ts_tablet_manager.cc:1403] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.010s
I20260709 13:06:16.665731 18529 raft_consensus.cc:359] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
W20260709 13:06:16.666527 18515 tablet.cc:2406] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:06:16.666630 18529 raft_consensus.cc:385] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.667173 18529 raft_consensus.cc:740] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.668469 18533 tablet_bootstrap.cc:654] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.668391 18529 consensus_queue.cc:260] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.673863 18530 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:16.674465 18530 ts_tablet_manager.cc:1403] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.029s	user 0.014s	sys 0.005s
I20260709 13:06:16.674487 18529 ts_tablet_manager.cc:1434] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:06:16.675670 18529 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.678426 18530 raft_consensus.cc:359] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.679231 18530 raft_consensus.cc:385] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.679572 18530 raft_consensus.cc:740] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.680472 18533 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:16.681118 18533 ts_tablet_manager.cc:1403] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.003s
I20260709 13:06:16.680576 18530 consensus_queue.cc:260] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.682636 18529 tablet_bootstrap.cc:654] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.683532 18530 ts_tablet_manager.cc:1434] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:06:16.684706 18530 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.684254 18533 raft_consensus.cc:359] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.685165 18533 raft_consensus.cc:385] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.685495 18533 raft_consensus.cc:740] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.686468 18533 consensus_queue.cc:260] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.692993 18533 ts_tablet_manager.cc:1434] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:06:16.694177 18533 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.705845 18533 tablet_bootstrap.cc:654] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.721637 18530 tablet_bootstrap.cc:654] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.725307 18533 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:16.725908 18533 ts_tablet_manager.cc:1403] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.008s
I20260709 13:06:16.729926 18529 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:16.729534 18533 raft_consensus.cc:359] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.730551 18533 raft_consensus.cc:385] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.730628 18529 ts_tablet_manager.cc:1403] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.055s	user 0.020s	sys 0.007s
I20260709 13:06:16.730940 18533 raft_consensus.cc:740] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.731901 18533 consensus_queue.cc:260] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.733501 18530 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:16.734242 18530 ts_tablet_manager.cc:1403] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.050s	user 0.019s	sys 0.001s
I20260709 13:06:16.734907 18533 ts_tablet_manager.cc:1434] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:16.734302 18529 raft_consensus.cc:359] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.735421 18529 raft_consensus.cc:385] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.735785 18529 raft_consensus.cc:740] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.736040 18533 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.736877 18529 consensus_queue.cc:260] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.738423 18530 raft_consensus.cc:359] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.739351 18530 raft_consensus.cc:385] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.739780 18530 raft_consensus.cc:740] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.740902 18530 consensus_queue.cc:260] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.742376 18529 ts_tablet_manager.cc:1434] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:06:16.743543 18533 tablet_bootstrap.cc:654] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.743599 18529 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.751893 18537 raft_consensus.cc:493] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.752959 18537 raft_consensus.cc:515] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.754937 18530 ts_tablet_manager.cc:1434] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 13:06:16.753607 18529 tablet_bootstrap.cc:654] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:16.765815 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:16.769762 18530 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.776913 18537 leader_election.cc:290] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:16.783020 18530 tablet_bootstrap.cc:654] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.788367 18533 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:16.789119 18533 ts_tablet_manager.cc:1403] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.053s	user 0.016s	sys 0.036s
I20260709 13:06:16.792948 18529 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:16.793759 18529 ts_tablet_manager.cc:1403] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.050s	user 0.018s	sys 0.009s
I20260709 13:06:16.792899 18533 raft_consensus.cc:359] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.794320 18533 raft_consensus.cc:385] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.794759 18533 raft_consensus.cc:740] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.795938 18537 raft_consensus.cc:493] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.796733 18537 raft_consensus.cc:515] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.795761 18533 consensus_queue.cc:260] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.797763 18530 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:16.797539 18529 raft_consensus.cc:359] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.798395 18530 ts_tablet_manager.cc:1403] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.009s
I20260709 13:06:16.798630 18529 raft_consensus.cc:385] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.796207 18535 raft_consensus.cc:493] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.799422 18535 raft_consensus.cc:515] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.800220 18537 leader_election.cc:290] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:16.802304 18530 raft_consensus.cc:359] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.803169 18530 raft_consensus.cc:385] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.803484 18530 raft_consensus.cc:740] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.804405 18530 consensus_queue.cc:260] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.808274 18533 ts_tablet_manager.cc:1434] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.019s	user 0.012s	sys 0.003s
I20260709 13:06:16.809743 18533 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.811699 18535 leader_election.cc:290] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:16.817510 18529 raft_consensus.cc:740] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.817858 18530 ts_tablet_manager.cc:1434] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.019s	user 0.010s	sys 0.000s
I20260709 13:06:16.818363 18533 tablet_bootstrap.cc:654] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.819119 18530 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.819617 18529 consensus_queue.cc:260] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.830029 18340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:16.831283 18340 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:16.831530 18530 tablet_bootstrap.cc:654] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.833011 18339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "087ed505256448bfafd264d7cb18396f" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:16.833113 18454 leader_election.cc:304] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
W20260709 13:06:16.834905 18454 leader_election.cc:343] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.836170 18533 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:16.837208 18533 ts_tablet_manager.cc:1403] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.012s
I20260709 13:06:16.836117 18537 raft_consensus.cc:2804] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.837782 18537 raft_consensus.cc:493] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.838275 18537 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.840925 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:16.842126 18489 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa72f0c464794888866a56f660990608 in term 0.
I20260709 13:06:16.848034 18529 ts_tablet_manager.cc:1434] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.054s	user 0.007s	sys 0.003s
I20260709 13:06:16.848469 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:16.848883 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:16.849382 18415 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:16.849753 18413 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa72f0c464794888866a56f660990608 in term 0.
I20260709 13:06:16.850454 18414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "087ed505256448bfafd264d7cb18396f" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:16.851004 18530 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:16.851224 18414 raft_consensus.cc:2468] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:16.851552 18304 leader_election.cc:304] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:16.852258 18537 raft_consensus.cc:515] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.853024 18535 raft_consensus.cc:2804] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.853188 18453 leader_election.cc:304] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: aa72f0c464794888866a56f660990608
I20260709 13:06:16.853602 18535 raft_consensus.cc:493] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.854303 18535 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.855446 18554 raft_consensus.cc:2804] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.855938 18554 raft_consensus.cc:493] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.856303 18554 raft_consensus.cc:3060] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.856664 18339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:16.849516 18529 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.857442 18339 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.851552 18530 ts_tablet_manager.cc:1403] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.004s
I20260709 13:06:16.865082 18529 tablet_bootstrap.cc:654] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.865504 18556 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.865849 18530 raft_consensus.cc:359] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.866200 18556 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.866770 18530 raft_consensus.cc:385] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.867127 18530 raft_consensus.cc:740] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.869460 18556 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:16.869573 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:16.870389 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
W20260709 13:06:16.871294 18304 leader_election.cc:343] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.871354 18489 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa72f0c464794888866a56f660990608 in term 0.
I20260709 13:06:16.873793 18339 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:16.873946 18306 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 31250b946a464780846c6ea7b435e870
I20260709 13:06:16.851553 18533 raft_consensus.cc:359] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.875152 18535 raft_consensus.cc:515] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.878458 18554 raft_consensus.cc:515] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.879220 18533 raft_consensus.cc:385] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.877446 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:16.879626 18533 raft_consensus.cc:740] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.880185 18556 raft_consensus.cc:2804] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.875485 18530 consensus_queue.cc:260] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.880620 18556 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.881287 18556 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.879932 18415 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.881932 18554 leader_election.cc:290] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:16.882259 18340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "087ed505256448bfafd264d7cb18396f" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:16.880594 18533 consensus_queue.cc:260] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.878592 18529 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:16.885486 18529 ts_tablet_manager.cc:1403] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.036s	user 0.007s	sys 0.014s
I20260709 13:06:16.886004 18533 ts_tablet_manager.cc:1434] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
W20260709 13:06:16.886072 18454 leader_election.cc:343] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.884569 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:16.887312 18413 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.888115 18533 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:16.889863 18535 leader_election.cc:290] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 election: Requested vote from peers 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:16.875025 18537 leader_election.cc:290] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:16.897255 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:16.886523 18414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "087ed505256448bfafd264d7cb18396f" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:16.890067 18529 raft_consensus.cc:359] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.897632 18454 leader_election.cc:304] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:16.898488 18414 raft_consensus.cc:3060] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.898715 18529 raft_consensus.cc:385] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.899101 18529 raft_consensus.cc:740] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.900041 18529 consensus_queue.cc:260] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.903196 18529 ts_tablet_manager.cc:1434] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260709 13:06:16.903800 18489 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.904469 18529 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.908602 18554 raft_consensus.cc:493] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.909327 18554 raft_consensus.cc:515] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.912810 18340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:16.913682 18340 raft_consensus.cc:2468] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:16.914676 18412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:16.915340 18454 leader_election.cc:304] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:16.915551 18412 raft_consensus.cc:2468] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:16.917721 18554 leader_election.cc:290] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:16.903781 18537 raft_consensus.cc:2804] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.918581 18554 raft_consensus.cc:2804] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.922914 18554 raft_consensus.cc:493] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.923444 18554 raft_consensus.cc:3060] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.923650 18537 raft_consensus.cc:697] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:16.928217 18414 raft_consensus.cc:2468] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:16.919391 18529 tablet_bootstrap.cc:654] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.928247 18489 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa72f0c464794888866a56f660990608 in term 1.
I20260709 13:06:16.922011 18556 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.935441 18453 leader_election.cc:304] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: aa72f0c464794888866a56f660990608
I20260709 13:06:16.936205 18306 leader_election.cc:304] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:16.937070 18535 raft_consensus.cc:2804] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.937474 18533 tablet_bootstrap.cc:654] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.937644 18535 raft_consensus.cc:697] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 LEADER]: Becoming Leader. State: Replica: aa72f0c464794888866a56f660990608, State: Running, Role: LEADER
I20260709 13:06:16.937592 18563 raft_consensus.cc:2804] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.939878 18535 consensus_queue.cc:237] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.940881 18530 ts_tablet_manager.cc:1434] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.079s	user 0.007s	sys 0.021s
I20260709 13:06:16.942188 18530 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:16.942754 18413 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa72f0c464794888866a56f660990608 in term 1.
I20260709 13:06:16.941778 18537 consensus_queue.cc:237] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.944412 18556 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 election: Requested vote from peers 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:16.946187 18562 raft_consensus.cc:493] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.946476 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:16.946791 18562 raft_consensus.cc:515] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
W20260709 13:06:16.947772 18304 leader_election.cc:343] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.948839 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:16.949585 18489 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.954173 18488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db070baeeb2b411fb19fed64cc792ab8" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:16.952446 18530 tablet_bootstrap.cc:654] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:16.955864 18562 leader_election.cc:290] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
W20260709 13:06:16.960072 18306 leader_election.cc:343] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.962342 18415 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:16.956964 18554 raft_consensus.cc:515] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.966893 18554 leader_election.cc:290] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:16.967803 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db070baeeb2b411fb19fed64cc792ab8" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:16.968072 18339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:16.968384 18529 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:16.962553 18563 raft_consensus.cc:697] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
W20260709 13:06:16.969506 18304 leader_election.cc:343] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.970077 18304 leader_election.cc:304] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa72f0c464794888866a56f660990608; no voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:16.970117 18563 consensus_queue.cc:237] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.969151 18529 ts_tablet_manager.cc:1403] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.065s	user 0.022s	sys 0.000s
I20260709 13:06:16.971582 18562 raft_consensus.cc:2749] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:16.969000 18339 raft_consensus.cc:3060] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.974253 18238 catalog_manager.cc:5697] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa72f0c464794888866a56f660990608 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "aa72f0c464794888866a56f660990608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:16.976055 18529 raft_consensus.cc:359] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.977129 18529 raft_consensus.cc:385] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.977541 18529 raft_consensus.cc:740] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.979485 18489 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa72f0c464794888866a56f660990608 in term 1.
I20260709 13:06:16.979266 18529 consensus_queue.cc:260] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.981251 18306 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 31250b946a464780846c6ea7b435e870
I20260709 13:06:16.982404 18529 ts_tablet_manager.cc:1434] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 13:06:16.983014 18562 raft_consensus.cc:2804] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.983578 18529 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:16.983574 18562 raft_consensus.cc:697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 LEADER]: Becoming Leader. State: Replica: aa72f0c464794888866a56f660990608, State: Running, Role: LEADER
I20260709 13:06:16.984439 18339 raft_consensus.cc:2468] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:16.984558 18562 consensus_queue.cc:237] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.985966 18454 leader_election.cc:304] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:16.990633 18554 raft_consensus.cc:2804] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.995046 18554 raft_consensus.cc:697] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:16.995467 18536 raft_consensus.cc:493] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.996172 18554 consensus_queue.cc:237] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.996307 18536 raft_consensus.cc:515] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:16.999897 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:17.000855 18413 raft_consensus.cc:3060] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.012560 18529 tablet_bootstrap.cc:654] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:17.022163 18536 leader_election.cc:290] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.025882 18533 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:17.026537 18533 ts_tablet_manager.cc:1403] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.139s	user 0.016s	sys 0.002s
I20260709 13:06:17.030827 18533 raft_consensus.cc:359] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.031625 18533 raft_consensus.cc:385] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.034276 18533 raft_consensus.cc:740] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.032142 18413 raft_consensus.cc:2468] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:17.038915 18537 raft_consensus.cc:493] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.039211 18533 consensus_queue.cc:260] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.039641 18537 raft_consensus.cc:515] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.045217 18533 ts_tablet_manager.cc:1434] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 13:06:17.045766 18339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:17.046478 18533 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:17.046636 18339 raft_consensus.cc:2468] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:17.046839 18530 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:17.047626 18530 ts_tablet_manager.cc:1403] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.106s	user 0.019s	sys 0.022s
I20260709 13:06:17.049518 18454 leader_election.cc:304] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:17.051120 18566 raft_consensus.cc:2804] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:17.051715 18566 raft_consensus.cc:493] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:17.052167 18566 raft_consensus.cc:3060] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.052151 18530 raft_consensus.cc:359] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.061151 18530 raft_consensus.cc:385] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.061403 18530 raft_consensus.cc:740] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.057472 18537 leader_election.cc:290] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.070070 18537 raft_consensus.cc:493] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.070672 18537 raft_consensus.cc:515] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.073136 18537 leader_election.cc:290] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.074242 18339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104363652bb743148f6c64f81c33060e" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:17.078837 18530 consensus_queue.cc:260] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.082434 18529 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:17.083056 18529 ts_tablet_manager.cc:1403] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.100s	user 0.015s	sys 0.001s
I20260709 13:06:17.083599 18339 raft_consensus.cc:2468] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:17.086191 18536 raft_consensus.cc:493] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.086791 18536 raft_consensus.cc:515] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.087935 18454 leader_election.cc:304] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:17.088956 18536 leader_election.cc:290] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.089161 18529 raft_consensus.cc:359] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.092975 18529 raft_consensus.cc:385] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.093258 18529 raft_consensus.cc:740] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.093580 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:17.094353 18413 raft_consensus.cc:2468] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:17.094133 18529 consensus_queue.cc:260] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.096339 18554 raft_consensus.cc:2804] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:17.089222 18566 raft_consensus.cc:515] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.099255 18530 ts_tablet_manager.cc:1434] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.051s	user 0.009s	sys 0.000s
I20260709 13:06:17.099977 18533 tablet_bootstrap.cc:654] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:17.101974 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104363652bb743148f6c64f81c33060e" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:17.102104 18537 raft_consensus.cc:493] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.102717 18413 raft_consensus.cc:2468] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:17.102881 18537 raft_consensus.cc:515] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.105532 18339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:17.106290 18339 raft_consensus.cc:2468] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 0.
I20260709 13:06:17.107342 18340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:17.109436 18530 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:17.111428 18529 ts_tablet_manager.cc:1434] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.028s	user 0.010s	sys 0.001s
I20260709 13:06:17.112839 18529 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:17.107370 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
W20260709 13:06:17.116114 18454 leader_election.cc:343] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:17.099252 18554 raft_consensus.cc:493] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:17.117728 18554 raft_consensus.cc:3060] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.117961 18380 leader_election.cc:304] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
W20260709 13:06:17.119577 18380 leader_election.cc:343] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:17.120555 18536 raft_consensus.cc:2804] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:17.108224 18537 leader_election.cc:290] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.121222 18536 raft_consensus.cc:493] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:17.122288 18536 raft_consensus.cc:3060] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.131462 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:17.132280 18413 raft_consensus.cc:2468] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:17.133508 18554 raft_consensus.cc:515] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.133826 18453 leader_election.cc:304] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: aa72f0c464794888866a56f660990608
I20260709 13:06:17.139778 18563 raft_consensus.cc:2804] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:17.140224 18563 raft_consensus.cc:493] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:17.140363 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104363652bb743148f6c64f81c33060e" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:17.141104 18338 raft_consensus.cc:3060] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.142704 18488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:17.142699 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:17.143962 18489 raft_consensus.cc:2393] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31250b946a464780846c6ea7b435e870 in current term 1: Already voted for candidate ac771fb495d84241b8a5dc7056ff239f in this term.
I20260709 13:06:17.140612 18563 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.152002 18340 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:17.152314 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104363652bb743148f6c64f81c33060e" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:17.153138 18413 raft_consensus.cc:3060] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.152202 18234 catalog_manager.cc:5697] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:17.152942 18340 raft_consensus.cc:3060] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.160426 18530 tablet_bootstrap.cc:654] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:17.163623 18234 catalog_manager.cc:5697] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:17.168649 18340 raft_consensus.cc:2468] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:17.174055 18529 tablet_bootstrap.cc:654] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:17.184074 18454 leader_election.cc:304] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:17.177086 18338 raft_consensus.cc:2468] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:17.177917 18566 leader_election.cc:290] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.179708 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:17.183220 18234 catalog_manager.cc:5697] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:17.151051 18554 leader_election.cc:290] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.173925 18413 raft_consensus.cc:2468] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:17.187772 18577 raft_consensus.cc:2804] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:17.189081 18577 raft_consensus.cc:697] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:17.190534 18577 consensus_queue.cc:237] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.192561 18454 leader_election.cc:304] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:17.197084 18536 raft_consensus.cc:515] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.201606 18566 raft_consensus.cc:2804] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:17.202498 18566 raft_consensus.cc:697] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:17.202891 18415 raft_consensus.cc:2393] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ac771fb495d84241b8a5dc7056ff239f in current term 1: Already voted for candidate 31250b946a464780846c6ea7b435e870 in this term.
I20260709 13:06:17.203511 18566 consensus_queue.cc:237] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.207975 18563 raft_consensus.cc:515] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.209455 18488 raft_consensus.cc:2393] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31250b946a464780846c6ea7b435e870 in current term 1: Already voted for candidate ac771fb495d84241b8a5dc7056ff239f in this term.
I20260709 13:06:17.210567 18563 leader_election.cc:290] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.211246 18562 raft_consensus.cc:493] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.211796 18562 raft_consensus.cc:515] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.212826 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:17.213675 18536 leader_election.cc:290] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.213892 18338 raft_consensus.cc:2393] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31250b946a464780846c6ea7b435e870 in current term 1: Already voted for candidate ac771fb495d84241b8a5dc7056ff239f in this term.
I20260709 13:06:17.216562 18529 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:17.212579 18237 catalog_manager.cc:5697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa72f0c464794888866a56f660990608 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "aa72f0c464794888866a56f660990608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:17.217124 18488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:17.218604 18562 leader_election.cc:290] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.217139 18529 ts_tablet_manager.cc:1403] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.105s	user 0.017s	sys 0.010s
I20260709 13:06:17.222998 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:17.218847 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
W20260709 13:06:17.223455 18306 leader_election.cc:343] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:17.220402 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
W20260709 13:06:17.225287 18454 leader_election.cc:343] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:17.227283 18304 leader_election.cc:343] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:17.227866 18304 leader_election.cc:304] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa72f0c464794888866a56f660990608; no voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:17.227900 18529 raft_consensus.cc:359] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.228915 18529 raft_consensus.cc:385] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.229310 18529 raft_consensus.cc:740] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.221248 18381 leader_election.cc:304] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31250b946a464780846c6ea7b435e870; no voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:17.221727 18530 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:17.229305 18562 raft_consensus.cc:2749] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:17.220347 18533 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:17.231045 18530 ts_tablet_manager.cc:1403] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.130s	user 0.023s	sys 0.003s
I20260709 13:06:17.230732 18529 consensus_queue.cc:260] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.231338 18533 ts_tablet_manager.cc:1403] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.185s	user 0.024s	sys 0.028s
I20260709 13:06:17.231911 18536 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.237468 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:17.240594 18530 raft_consensus.cc:359] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.241809 18530 raft_consensus.cc:385] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.241852 18381 leader_election.cc:304] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31250b946a464780846c6ea7b435e870; no voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:17.242373 18530 raft_consensus.cc:740] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.243078 18533 raft_consensus.cc:359] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.243917 18533 raft_consensus.cc:385] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.244287 18533 raft_consensus.cc:740] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.244015 18530 consensus_queue.cc:260] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.245286 18533 consensus_queue.cc:260] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.247491 18530 ts_tablet_manager.cc:1434] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.000s
I20260709 13:06:17.247493 18533 ts_tablet_manager.cc:1434] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 13:06:17.248955 18533 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f: Bootstrap starting.
I20260709 13:06:17.251834 18530 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:17.254763 18529 ts_tablet_manager.cc:1434] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.032s	user 0.011s	sys 0.017s
I20260709 13:06:17.256062 18529 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870: Bootstrap starting.
I20260709 13:06:17.256633 18585 raft_consensus.cc:2749] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:17.260442 18536 raft_consensus.cc:2749] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:17.262149 18529 tablet_bootstrap.cc:654] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:17.265894 18234 catalog_manager.cc:5697] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:17.268890 18234 catalog_manager.cc:5697] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:17.268826 18415 raft_consensus.cc:2468] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:17.276609 18530 tablet_bootstrap.cc:654] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:17.280076 18585 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.280705 18585 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.283237 18585 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.285581 18533 tablet_bootstrap.cc:654] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:17.288834 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:17.296782 18562 raft_consensus.cc:493] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.297621 18562 raft_consensus.cc:515] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
W20260709 13:06:17.299254 18381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:37131, user_credentials={real_user=slave}} blocked reactor thread for 57517.5us
I20260709 13:06:17.300347 18453 leader_election.cc:304] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: aa72f0c464794888866a56f660990608
I20260709 13:06:17.301095 18566 raft_consensus.cc:2804] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:17.301682 18566 raft_consensus.cc:697] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:17.302039 18562 leader_election.cc:290] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.302630 18566 consensus_queue.cc:237] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.303267 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:17.303678 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "aa72f0c464794888866a56f660990608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:17.304481 18415 raft_consensus.cc:2393] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa72f0c464794888866a56f660990608 in current term 1: Already voted for candidate ac771fb495d84241b8a5dc7056ff239f in this term.
I20260709 13:06:17.305974 18304 leader_election.cc:304] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa72f0c464794888866a56f660990608; no voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:17.306541 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:17.307735 18489 raft_consensus.cc:2393] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31250b946a464780846c6ea7b435e870 in current term 1: Already voted for candidate aa72f0c464794888866a56f660990608 in this term.
I20260709 13:06:17.307652 18562 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.318635 18533 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f: No bootstrap required, opened a new log
I20260709 13:06:17.318238 18530 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: No bootstrap required, opened a new log
I20260709 13:06:17.319523 18533 ts_tablet_manager.cc:1403] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f: Time spent bootstrapping tablet: real 0.071s	user 0.010s	sys 0.002s
I20260709 13:06:17.319680 18530 ts_tablet_manager.cc:1403] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.068s	user 0.018s	sys 0.001s
I20260709 13:06:17.322932 18530 raft_consensus.cc:359] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.323364 18529 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870: No bootstrap required, opened a new log
I20260709 13:06:17.323587 18530 raft_consensus.cc:385] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.318143 18381 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31250b946a464780846c6ea7b435e870; no voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:17.323954 18529 ts_tablet_manager.cc:1403] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870: Time spent bootstrapping tablet: real 0.068s	user 0.036s	sys 0.029s
I20260709 13:06:17.323913 18530 raft_consensus.cc:740] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.324879 18586 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.325278 18530 consensus_queue.cc:260] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.327571 18562 raft_consensus.cc:2749] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:17.329958 18533 raft_consensus.cc:359] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.330310 18529 raft_consensus.cc:359] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.331070 18529 raft_consensus.cc:385] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.331353 18529 raft_consensus.cc:740] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.332110 18529 consensus_queue.cc:260] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.330724 18533 raft_consensus.cc:385] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:17.332913 18586 raft_consensus.cc:2749] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:17.333524 18533 raft_consensus.cc:740] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Initialized, Role: FOLLOWER
I20260709 13:06:17.336954 18533 consensus_queue.cc:260] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.334959 18529 ts_tablet_manager.cc:1434] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 13:06:17.339148 18533 ts_tablet_manager.cc:1434] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.004s
I20260709 13:06:17.335589 18530 ts_tablet_manager.cc:1434] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:06:17.343511 18237 catalog_manager.cc:5697] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:17.354211 18566 consensus_queue.cc:1048] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.371284 18577 consensus_queue.cc:1048] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.411242 18586 raft_consensus.cc:493] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.411932 18586 raft_consensus.cc:515] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.413915 18586 leader_election.cc:290] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.414880 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db070baeeb2b411fb19fed64cc792ab8" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:17.415126 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db070baeeb2b411fb19fed64cc792ab8" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:17.415612 18338 raft_consensus.cc:2468] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 0.
I20260709 13:06:17.415836 18489 raft_consensus.cc:2468] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 0.
I20260709 13:06:17.416734 18380 leader_election.cc:304] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:17.417518 18586 raft_consensus.cc:2804] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:17.417918 18586 raft_consensus.cc:493] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:17.418203 18586 raft_consensus.cc:3060] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.423341 18586 raft_consensus.cc:515] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.425221 18586 leader_election.cc:290] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:17.426201 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db070baeeb2b411fb19fed64cc792ab8" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:17.426337 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db070baeeb2b411fb19fed64cc792ab8" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:17.426864 18338 raft_consensus.cc:3060] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.426991 18489 raft_consensus.cc:3060] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.428790 18562 consensus_queue.cc:1048] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.434216 18338 raft_consensus.cc:2468] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 1.
I20260709 13:06:17.434636 18489 raft_consensus.cc:2468] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 1.
I20260709 13:06:17.435453 18380 leader_election.cc:304] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:17.436442 18586 raft_consensus.cc:2804] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:17.436949 18586 raft_consensus.cc:697] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 1 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:17.437822 18586 consensus_queue.cc:237] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.443629 18562 consensus_queue.cc:1048] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.451197 18236 catalog_manager.cc:5697] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:17.454483 18577 consensus_queue.cc:1048] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.464808 18338 raft_consensus.cc:3060] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.473518 18566 consensus_queue.cc:1048] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.484059 18566 raft_consensus.cc:493] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:17.484560 18566 raft_consensus.cc:515] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.486771 18566 leader_election.cc:290] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.487844 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:17.487926 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608" is_pre_election: true
I20260709 13:06:17.488564 18415 raft_consensus.cc:2468] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:17.488678 18338 raft_consensus.cc:2468] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 0.
I20260709 13:06:17.489729 18453 leader_election.cc:304] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:17.490926 18566 raft_consensus.cc:2804] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:17.491308 18566 raft_consensus.cc:493] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:17.491645 18566 raft_consensus.cc:3060] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.497819 18566 raft_consensus.cc:515] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.499650 18566 leader_election.cc:290] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:17.500547 18338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:17.500720 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:17.501554 18338 raft_consensus.cc:3060] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.501605 18415 raft_consensus.cc:3060] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.506862 18338 raft_consensus.cc:2468] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:17.507177 18415 raft_consensus.cc:2468] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:17.507983 18454 leader_election.cc:304] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:17.508889 18566 raft_consensus.cc:2804] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:17.509394 18566 raft_consensus.cc:697] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:17.510185 18566 consensus_queue.cc:237] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:17.518239 18236 catalog_manager.cc:5697] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:17.532755 18566 consensus_queue.cc:1048] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.543792 18554 consensus_queue.cc:1048] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.555327 18562 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.569165 18562 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.577507 18563 consensus_queue.cc:1048] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.591321 18537 consensus_queue.cc:1048] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.615752 18537 consensus_queue.cc:1048] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.627579 18537 consensus_queue.cc:1048] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.741915 18563 consensus_queue.cc:1048] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.756798 18554 consensus_queue.cc:1048] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.758955  7481 ts_manager.cc:295] Set tserver state for aa72f0c464794888866a56f660990608 to MAINTENANCE_MODE
I20260709 13:06:17.760021  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
W20260709 13:06:17.767320 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:17.802088  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:17.803350  7481 tablet_replica.cc:333] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.804106  7481 raft_consensus.cc:2243] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.804649  7481 raft_consensus.cc:2272] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.807473  7481 tablet_replica.cc:333] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.808259  7481 raft_consensus.cc:2243] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.808806  7481 raft_consensus.cc:2272] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.811297  7481 tablet_replica.cc:333] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.811910  7481 raft_consensus.cc:2243] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.812381  7481 raft_consensus.cc:2272] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.814203  7481 tablet_replica.cc:333] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.814760  7481 raft_consensus.cc:2243] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.815209  7481 raft_consensus.cc:2272] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.818274  7481 tablet_replica.cc:333] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.818881  7481 raft_consensus.cc:2243] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.819386  7481 raft_consensus.cc:2272] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:17.821102 18380 consensus_peers.cc:597] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:17.822569  7481 tablet_replica.cc:333] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.823189  7481 raft_consensus.cc:2243] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.823576  7481 raft_consensus.cc:2272] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.825804  7481 tablet_replica.cc:333] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.826326  7481 raft_consensus.cc:2243] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.826793  7481 raft_consensus.cc:2272] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.829135  7481 tablet_replica.cc:333] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.829727  7481 raft_consensus.cc:2243] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:17.830091  7481 raft_consensus.cc:2272] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.831787  7481 tablet_replica.cc:333] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.832234  7481 raft_consensus.cc:2243] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:17.833005  7481 raft_consensus.cc:2272] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.835089  7481 tablet_replica.cc:333] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: stopping tablet replica
I20260709 13:06:17.835544  7481 raft_consensus.cc:2243] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:17.836278  7481 raft_consensus.cc:2272] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:17.868371  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
I20260709 13:06:17.913646 18554 consensus_queue.cc:1048] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.918664 18586 consensus_queue.cc:1048] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:06:17.936527 18454 consensus_peers.cc:597] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:17.937234 18454 consensus_peers.cc:597] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:17.946163 18454 consensus_peers.cc:597] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:17.958361 18454 consensus_peers.cc:597] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.101300 18454 consensus_peers.cc:597] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.189091 18454 consensus_peers.cc:597] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.235968 18454 consensus_peers.cc:597] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.638157 18454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20260709 13:06:18.768426 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:18.941538 18617 raft_consensus.cc:493] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa72f0c464794888866a56f660990608)
I20260709 13:06:18.942214 18617 raft_consensus.cc:515] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:18.943982 18617 leader_election.cc:290] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:18.945150 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
W20260709 13:06:18.948519 18380 leader_election.cc:336] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111)
I20260709 13:06:18.949056 18380 leader_election.cc:304] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31250b946a464780846c6ea7b435e870; no voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:18.949426 18537 raft_consensus.cc:493] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa72f0c464794888866a56f660990608)
I20260709 13:06:18.950004 18537 raft_consensus.cc:515] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:18.950441 18617 raft_consensus.cc:2749] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:18.952646 18537 leader_election.cc:290] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:18.954542 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:18.955303 18415 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:18.956493 18453 leader_election.cc:304] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: 
W20260709 13:06:18.957082 18454 leader_election.cc:336] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111)
I20260709 13:06:18.957327 18537 raft_consensus.cc:2804] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:18.957851 18537 raft_consensus.cc:493] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa72f0c464794888866a56f660990608)
I20260709 13:06:18.958216 18537 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:18.964711 18537 raft_consensus.cc:515] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:18.966572 18537 leader_election.cc:290] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:18.967856 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:18.968678 18415 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:18.971282 18454 leader_election.cc:336] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111)
I20260709 13:06:18.975934 18415 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 2.
I20260709 13:06:18.977623 18453 leader_election.cc:304] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: aa72f0c464794888866a56f660990608
I20260709 13:06:18.978665 18537 raft_consensus.cc:2804] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:18.979140 18537 raft_consensus.cc:697] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:18.979863 18537 consensus_queue.cc:237] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:18.987751 18237 catalog_manager.cc:5697] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 1 to 2, leader changed from aa72f0c464794888866a56f660990608 (127.7.78.65) to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 2 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:19.066695 18617 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa72f0c464794888866a56f660990608)
I20260709 13:06:19.067274 18617 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:19.069056 18617 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:19.070191 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac771fb495d84241b8a5dc7056ff239f" is_pre_election: true
I20260709 13:06:19.072719 18537 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa72f0c464794888866a56f660990608)
W20260709 13:06:19.073148 18380 leader_election.cc:336] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111)
I20260709 13:06:19.073272 18537 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:19.073792 18380 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31250b946a464780846c6ea7b435e870; no voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:19.074565 18617 raft_consensus.cc:2749] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:19.075634 18537 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:19.076885 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870" is_pre_election: true
I20260709 13:06:19.077634 18415 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 1.
I20260709 13:06:19.078797 18453 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:19.079506 18537 raft_consensus.cc:2804] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:06:19.079581 18454 leader_election.cc:336] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111)
I20260709 13:06:19.079851 18537 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa72f0c464794888866a56f660990608)
I20260709 13:06:19.080243 18537 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:19.085999 18537 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:19.087852 18537 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:19.089149 18415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:19.089823 18415 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:19.091985 18454 leader_election.cc:336] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111)
I20260709 13:06:19.095526 18415 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 2.
I20260709 13:06:19.096864 18453 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: aa72f0c464794888866a56f660990608
I20260709 13:06:19.097761 18537 raft_consensus.cc:2804] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:19.098383 18537 raft_consensus.cc:697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:19.099259 18537 consensus_queue.cc:237] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:19.107121 18237 catalog_manager.cc:5697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 1 to 2, leader changed from aa72f0c464794888866a56f660990608 (127.7.78.65) to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 2 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:19.417126 18454 consensus_peers.cc:597] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.458312 18415 raft_consensus.cc:1275] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:19.459705 18537 consensus_queue.cc:1048] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:19.604825 18415 raft_consensus.cc:1275] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:19.606258 18537 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:06:19.628892 18454 consensus_peers.cc:597] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.769598 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:06:20.112701 18380 consensus_peers.cc:597] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.285467 18454 consensus_peers.cc:597] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.353255 18454 consensus_peers.cc:597] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.394302 18454 consensus_peers.cc:597] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.430830 18454 consensus_peers.cc:597] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.595026 18454 consensus_peers.cc:597] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.600553 18454 consensus_peers.cc:597] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.751989 18454 consensus_peers.cc:597] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:20.770736 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:06:21.771867 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:06:22.048883 18454 consensus_peers.cc:597] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:22.066735 18454 consensus_peers.cc:597] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:22.570869 18380 consensus_peers.cc:597] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:22.772881 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:06:22.828039 18454 consensus_peers.cc:597] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:22.888989  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:22.895725 18642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:22.896773 18643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:22.901163 18645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:22.901607  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:22.903147  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:22.903420  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:22.903612  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602382903588 us; error 0 us; skew 500 ppm
I20260709 13:06:22.904348  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:22.907735  7481 webserver.cc:533] Webserver started at http://127.7.78.65:40309/ using document root <none> and password file <none>
I20260709 13:06:22.908412  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:22.908684  7481 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 13:06:22.912719  7481 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 13:06:22.917647 18650 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:22.918708  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
I20260709 13:06:22.919061  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "aa72f0c464794888866a56f660990608"
format_stamp: "Formatted at 2026-07-09 13:06:15 on dist-test-slave-50cs"
I20260709 13:06:22.919358  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:22.943269  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:22.944422  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:22.946023  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:22.949932 18657 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
W20260709 13:06:22.984126 18454 consensus_peers.cc:597] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:22.991053 18454 consensus_peers.cc:597] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:23.006071 18454 consensus_peers.cc:597] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:23.017982  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 13:06:23.018393  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.070s	user 0.000s	sys 0.003s
I20260709 13:06:23.019028  7481 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 13:06:23.024386 18657 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.037245 18657 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.038414 18657 tablet_bootstrap.cc:492] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.039393 18657 ts_tablet_manager.cc:1403] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.011s
I20260709 13:06:23.042335 18657 raft_consensus.cc:359] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.043195 18657 raft_consensus.cc:740] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.044166 18657 consensus_queue.cc:260] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.067238 18657 ts_tablet_manager.cc:1434] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.019s
I20260709 13:06:23.068277 18657 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Bootstrap starting.
W20260709 13:06:23.070735 18454 consensus_peers.cc:597] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:23.086371 18657 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.087214  7481 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 13:06:23.087505 18657 tablet_bootstrap.cc:492] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.087589  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.069s	user 0.065s	sys 0.001s
I20260709 13:06:23.088441 18657 ts_tablet_manager.cc:1403] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.004s
I20260709 13:06:23.091223 18657 raft_consensus.cc:359] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.091997 18657 raft_consensus.cc:740] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.092739 18657 consensus_queue.cc:260] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.094966 18657 ts_tablet_manager.cc:1434] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:06:23.096053 18657 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.114341 18657 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.115274 18657 tablet_bootstrap.cc:492] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.116040 18657 ts_tablet_manager.cc:1403] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.020s	user 0.017s	sys 0.000s
I20260709 13:06:23.118433 18657 raft_consensus.cc:359] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.119087 18657 raft_consensus.cc:740] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.119845 18657 consensus_queue.cc:260] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.121970 18657 ts_tablet_manager.cc:1434] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.003s
I20260709 13:06:23.122816 18657 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.137315 18657 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.138571 18657 tablet_bootstrap.cc:492] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.139362 18657 ts_tablet_manager.cc:1403] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.017s	user 0.014s	sys 0.000s
I20260709 13:06:23.142270 18657 raft_consensus.cc:359] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.143123 18657 raft_consensus.cc:740] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.143930 18657 consensus_queue.cc:260] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.146234 18657 ts_tablet_manager.cc:1434] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:06:23.147245 18657 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.148852  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:45805
I20260709 13:06:23.148983 18723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:45805 every 8 connection(s)
I20260709 13:06:23.154310  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:23.163743 18657 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.164834 18657 tablet_bootstrap.cc:492] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.165573 18657 ts_tablet_manager.cc:1403] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.004s
I20260709 13:06:23.168358 18657 raft_consensus.cc:359] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.169009 18657 raft_consensus.cc:740] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.169767 18657 consensus_queue.cc:260] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.171727 18657 ts_tablet_manager.cc:1434] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:06:23.173169 18657 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.173604 18724 heartbeater.cc:344] Connected to a master server at 127.7.78.126:43571
I20260709 13:06:23.174100 18724 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:23.175411 18724 heartbeater.cc:507] Master 127.7.78.126:43571 requested a full tablet report, sending...
I20260709 13:06:23.178894 18489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:23.179713 18489 raft_consensus.cc:606] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:23.182211 18236 ts_manager.cc:194] Re-registered known tserver with Master: aa72f0c464794888866a56f660990608 (127.7.78.65:45805)
I20260709 13:06:23.182999 18489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "087ed505256448bfafd264d7cb18396f"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:23.183722 18489 raft_consensus.cc:606] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:23.186868 18489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:23.187474 18489 raft_consensus.cc:606] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:23.190501 18489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:23.191073 18489 raft_consensus.cc:606] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:23.193544 18489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:23.193894 18236 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38280
I20260709 13:06:23.194250 18489 raft_consensus.cc:606] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:23.194723 18657 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.195773 18657 tablet_bootstrap.cc:492] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.195835  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:06:23.196508 18657 ts_tablet_manager.cc:1403] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.000s
I20260709 13:06:23.196552  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:23.197237 18724 heartbeater.cc:499] Master 127.7.78.126:43571 was elected leader, sending a full tablet report...
I20260709 13:06:23.197520  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:23.199250 18657 raft_consensus.cc:359] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.199997 18657 raft_consensus.cc:740] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.200824 18657 consensus_queue.cc:260] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.203073 18657 ts_tablet_manager.cc:1434] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:06:23.203948 18657 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.219172 18657 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.220232 18657 tablet_bootstrap.cc:492] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.221024 18657 ts_tablet_manager.cc:1403] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 13:06:23.223863 18657 raft_consensus.cc:359] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.224651 18657 raft_consensus.cc:740] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.225476 18657 consensus_queue.cc:260] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.227483 18657 ts_tablet_manager.cc:1434] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:06:23.228266 18657 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.241532 18657 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.242323 18657 tablet_bootstrap.cc:492] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.242988 18657 ts_tablet_manager.cc:1403] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.004s
I20260709 13:06:23.244994 18657 raft_consensus.cc:359] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.245678 18657 raft_consensus.cc:740] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.246338 18657 consensus_queue.cc:260] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.248140 18657 ts_tablet_manager.cc:1434] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:06:23.248978 18657 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.262722 18657 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.263688 18657 tablet_bootstrap.cc:492] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.264377 18657 ts_tablet_manager.cc:1403] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.000s
I20260709 13:06:23.266739 18657 raft_consensus.cc:359] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.267334 18657 raft_consensus.cc:740] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.267946 18657 consensus_queue.cc:260] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.269884 18657 ts_tablet_manager.cc:1434] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 13:06:23.270937 18657 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Bootstrap starting.
I20260709 13:06:23.283900 18657 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.284691 18657 tablet_bootstrap.cc:492] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Bootstrap complete.
I20260709 13:06:23.285247 18657 ts_tablet_manager.cc:1403] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Time spent bootstrapping tablet: real 0.015s	user 0.010s	sys 0.000s
I20260709 13:06:23.287094 18657 raft_consensus.cc:359] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.287611 18657 raft_consensus.cc:740] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa72f0c464794888866a56f660990608, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.288198 18657 consensus_queue.cc:260] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.289898 18657 ts_tablet_manager.cc:1434] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:06:23.324223 18635 raft_consensus.cc:993] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.324782 18659 raft_consensus.cc:1081] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.326540 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:23.327122 18413 raft_consensus.cc:493] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:23.327410 18413 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.333668 18413 raft_consensus.cc:515] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.335680 18413 leader_election.cc:290] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:23.337478 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:23.338138 18489 raft_consensus.cc:3055] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:23.338460 18489 raft_consensus.cc:740] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:23.339275 18489 consensus_queue.cc:260] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.340373 18489 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.345856 18698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:23.346693 18698 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.347213 18489 raft_consensus.cc:2468] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.348712 18381 leader_election.cc:304] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:23.349622 18617 raft_consensus.cc:2804] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:23.351135 18617 raft_consensus.cc:697] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:23.352212 18617 consensus_queue.cc:237] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.353645 18698 raft_consensus.cc:2468] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.360766 18236 catalog_manager.cc:5697] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 1 to 2, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:23.394754 18698 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.473059 18659 raft_consensus.cc:993] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.473585 18635 raft_consensus.cc:1081] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.475144 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:23.475603 18413 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:23.475868 18413 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:23.481518 18413 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.483232 18413 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 3 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:23.484236 18698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:23.484524 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:23.484823 18698 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:06:23.485220 18489 raft_consensus.cc:3055] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:23.485670 18489 raft_consensus.cc:740] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:23.486586 18489 consensus_queue.cc:260] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.487679 18489 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:23.491012 18698 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 3.
I20260709 13:06:23.492309 18380 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:23.493257 18617 raft_consensus.cc:2804] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:23.493832 18489 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 3.
I20260709 13:06:23.493855 18617 raft_consensus.cc:697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:23.495064 18617 consensus_queue.cc:237] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.502838 18236 catalog_manager.cc:5697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 2 to 3, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:23.539460 18659 raft_consensus.cc:993] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.539984 18635 raft_consensus.cc:1081] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.541756 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "087ed505256448bfafd264d7cb18396f"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:23.542317 18413 raft_consensus.cc:493] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:23.542654 18413 raft_consensus.cc:3060] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.548288 18413 raft_consensus.cc:515] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.550448 18413 leader_election.cc:290] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:23.551548 18698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "087ed505256448bfafd264d7cb18396f" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:23.551627 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "087ed505256448bfafd264d7cb18396f" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:23.552220 18698 raft_consensus.cc:3060] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.552381 18489 raft_consensus.cc:3055] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:23.552896 18489 raft_consensus.cc:740] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:23.553617 18489 consensus_queue.cc:260] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.554579 18489 raft_consensus.cc:3060] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.558135 18698 raft_consensus.cc:2468] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.559443 18380 leader_election.cc:304] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:23.559669 18489 raft_consensus.cc:2468] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.560425 18617 raft_consensus.cc:2804] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:23.560956 18617 raft_consensus.cc:697] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:23.561942 18617 consensus_queue.cc:237] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.564162 18659 raft_consensus.cc:993] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.564705 18635 raft_consensus.cc:1081] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.566656 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:23.567639 18413 raft_consensus.cc:493] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:23.567947 18413 raft_consensus.cc:3060] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.570648 18236 catalog_manager.cc:5697] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 1 to 2, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:23.573289 18413 raft_consensus.cc:515] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.575778 18413 leader_election.cc:290] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:23.575945 18698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:23.576632 18698 raft_consensus.cc:3060] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.577028 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db35f4181aa46adb426b1bae078a0df" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:23.577692 18489 raft_consensus.cc:3055] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:23.578006 18489 raft_consensus.cc:740] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:23.578835 18489 consensus_queue.cc:260] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.580190 18489 raft_consensus.cc:3060] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.584285 18698 raft_consensus.cc:2468] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.585572 18380 leader_election.cc:304] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:23.586400 18617 raft_consensus.cc:2804] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:23.586853 18617 raft_consensus.cc:697] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:23.586920 18659 raft_consensus.cc:993] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.587576 18635 raft_consensus.cc:1081] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:23.587800 18617 consensus_queue.cc:237] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.587782 18489 raft_consensus.cc:2468] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.589278 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:23.590199 18413 raft_consensus.cc:493] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:23.590644 18413 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.596024 18236 catalog_manager.cc:5697] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 1 to 2, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:23.596488 18413 raft_consensus.cc:515] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.598840 18413 leader_election.cc:290] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:23.599521 18698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:23.600010 18489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:23.600163 18698 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.603181 18489 raft_consensus.cc:3055] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:23.603619 18489 raft_consensus.cc:740] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:23.604549 18489 consensus_queue.cc:260] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.605986 18489 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:23.610677 18698 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.613742 18380 leader_election.cc:304] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:23.614221 18489 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:23.614604 18617 raft_consensus.cc:2804] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:23.615384 18617 raft_consensus.cc:697] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:23.616185 18617 consensus_queue.cc:237] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:23.625717 18236 catalog_manager.cc:5697] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 1 to 2, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:23.773936 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:23.867591 18698 raft_consensus.cc:1275] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:23.869248 18617 consensus_queue.cc:1048] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:23.884024 18489 raft_consensus.cc:1275] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:23.886364 18617 consensus_queue.cc:1048] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:24.003415 18489 raft_consensus.cc:1275] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:24.004784 18488 raft_consensus.cc:1275] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:24.004803 18617 consensus_queue.cc:1048] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:24.007503 18753 consensus_queue.cc:1048] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:24.007910 18698 raft_consensus.cc:1275] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:24.009454 18753 consensus_queue.cc:1048] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:24.016511 18697 raft_consensus.cc:1275] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:24.018494 18753 consensus_queue.cc:1048] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:24.030299 18487 raft_consensus.cc:1275] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:24.035873 18737 consensus_queue.cc:1048] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:24.038630 18698 raft_consensus.cc:1275] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:24.040657 18753 consensus_queue.cc:1048] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:24.060259 18697 raft_consensus.cc:1275] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:24.062991 18762 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:24.080742 18487 raft_consensus.cc:1275] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:24.082221 18617 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:24.208243  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:24.228277 18413 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a"
dest_uuid: "31250b946a464780846c6ea7b435e870"
mode: GRACEFUL
new_leader_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:59698
I20260709 13:06:24.228835 18413 raft_consensus.cc:606] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Received request to transfer leadership to TS ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:24.231812 18413 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51"
dest_uuid: "31250b946a464780846c6ea7b435e870"
mode: GRACEFUL
new_leader_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:59698
I20260709 13:06:24.232352 18413 raft_consensus.cc:606] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Received request to transfer leadership to TS ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:24.233593  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:24.234311  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:24.234997  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:24.407876 18753 raft_consensus.cc:993] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870: : Instructing follower ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:24.408483 18762 raft_consensus.cc:1081] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Signalling peer ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:24.410277 18487 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:47274
I20260709 13:06:24.410885 18487 raft_consensus.cc:493] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:24.411289 18487 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:24.418531 18487 raft_consensus.cc:515] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:24.420372 18487 leader_election.cc:290] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 3 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:24.421365 18697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:24.422006 18697 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:24.424551 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea09119560864fdabe6de1bde69b8f51" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:24.425244 18413 raft_consensus.cc:3055] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:24.425566 18413 raft_consensus.cc:740] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:24.426330 18413 consensus_queue.cc:260] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:24.427349 18413 raft_consensus.cc:3060] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:24.427520 18697 raft_consensus.cc:2468] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 3.
I20260709 13:06:24.428757 18454 leader_election.cc:304] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:24.429601 18755 raft_consensus.cc:2804] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:24.430034 18755 raft_consensus.cc:697] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 3 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:24.430832 18755 consensus_queue.cc:237] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:24.433847 18413 raft_consensus.cc:2468] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 3.
I20260709 13:06:24.437776 18236 catalog_manager.cc:5697] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 2 to 3, leader changed from 31250b946a464780846c6ea7b435e870 (127.7.78.66) to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 3 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:24.507401 18753 raft_consensus.cc:993] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870: : Instructing follower ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:24.507995 18753 raft_consensus.cc:1081] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Signalling peer ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:24.509618 18487 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:47274
I20260709 13:06:24.510424 18487 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:24.510756 18487 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:24.515725 18487 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:24.517701 18487 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 4 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:24.518896 18697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:24.518877 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:24.519816 18697 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:24.519891 18413 raft_consensus.cc:3055] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:24.520366 18413 raft_consensus.cc:740] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:24.521049 18413 consensus_queue.cc:260] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:24.522011 18413 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:24.525571 18697 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 4.
I20260709 13:06:24.526683 18413 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 4.
I20260709 13:06:24.526767 18454 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:24.527598 18755 raft_consensus.cc:2804] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:24.528041 18755 raft_consensus.cc:697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:24.528764 18755 consensus_queue.cc:237] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:24.535251 18236 catalog_manager.cc:5697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 3 to 4, leader changed from 31250b946a464780846c6ea7b435e870 (127.7.78.66) to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 4 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:24.775092 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:24.846676 18413 raft_consensus.cc:1275] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:24.848106 18659 consensus_queue.cc:1048] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:24.859406 18697 raft_consensus.cc:1275] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:24.860752 18773 consensus_queue.cc:1048] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:25.099473 18413 raft_consensus.cc:1275] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:25.100692 18773 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:25.109699 18697 raft_consensus.cc:1275] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 4 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:25.111411 18659 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:25.241560  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:25.249202 18487 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:25.249743 18487 raft_consensus.cc:606] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:25.252537 18487 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:25.253130 18487 raft_consensus.cc:606] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:25.254397  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:25.254873  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:25.255503  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:25.324554 18755 raft_consensus.cc:993] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:25.325723 18659 raft_consensus.cc:1081] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:25.328186 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:25.328796 18413 raft_consensus.cc:493] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:25.329134 18413 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:25.339773 18413 raft_consensus.cc:515] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:25.341794 18413 leader_election.cc:290] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 3 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:25.343413 18487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:25.343339 18697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:25.344599 18487 raft_consensus.cc:3055] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:25.344833 18697 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:25.344973 18487 raft_consensus.cc:740] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:25.346486 18487 consensus_queue.cc:260] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:25.348939 18487 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:25.355809 18697 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 3.
I20260709 13:06:25.357340 18380 leader_election.cc:304] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:25.358367 18737 raft_consensus.cc:2804] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:25.358719 18487 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 3.
I20260709 13:06:25.359522 18737 raft_consensus.cc:697] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:25.361116 18737 consensus_queue.cc:237] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:25.371621 18237 catalog_manager.cc:5697] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 2 to 3, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:25.589342 18755 raft_consensus.cc:993] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:25.590013 18659 raft_consensus.cc:1081] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:25.591794 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:25.592550 18413 raft_consensus.cc:493] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:25.593132 18413 raft_consensus.cc:3060] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:25.599448 18413 raft_consensus.cc:515] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:25.600948 18413 leader_election.cc:290] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:25.602135 18487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:25.602253 18697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d9d73a52af14ff69e7996ebcc71131d" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:25.602937 18487 raft_consensus.cc:3055] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:25.603216 18697 raft_consensus.cc:3060] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:25.603444 18487 raft_consensus.cc:740] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:25.604948 18487 consensus_queue.cc:260] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:25.606739 18487 raft_consensus.cc:3060] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:25.616104 18697 raft_consensus.cc:2468] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:25.617718 18380 leader_election.cc:304] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:25.617739 18487 raft_consensus.cc:2468] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:25.618794 18737 raft_consensus.cc:2804] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:25.619324 18737 raft_consensus.cc:697] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:25.620364 18737 consensus_queue.cc:237] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:25.628600 18237 catalog_manager.cc:5697] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 1 to 2, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:25.776415 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:25.843032 18487 raft_consensus.cc:1275] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:25.844942 18780 consensus_queue.cc:1048] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:25.857376 18697 raft_consensus.cc:1275] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:25.859169 18737 consensus_queue.cc:1048] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:26.010465 18487 raft_consensus.cc:1275] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:26.012056 18737 consensus_queue.cc:1048] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:26.022715 18697 raft_consensus.cc:1275] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:26.024348 18737 consensus_queue.cc:1048] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:26.265727  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:26.274569 18413 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3"
dest_uuid: "31250b946a464780846c6ea7b435e870"
mode: GRACEFUL
new_leader_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:59698
I20260709 13:06:26.275184 18413 raft_consensus.cc:606] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Received request to transfer leadership to TS ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:26.278097 18413 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf"
dest_uuid: "31250b946a464780846c6ea7b435e870"
mode: GRACEFUL
new_leader_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:59698
I20260709 13:06:26.278662 18413 raft_consensus.cc:606] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Received request to transfer leadership to TS ac771fb495d84241b8a5dc7056ff239f
I20260709 13:06:26.279662  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:26.280092  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:26.280977  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:26.480978 18788 raft_consensus.cc:993] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870: : Instructing follower ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:26.481592 18788 raft_consensus.cc:1081] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Signalling peer ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:26.484367 18487 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:47274
I20260709 13:06:26.484982 18487 raft_consensus.cc:493] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:26.485360 18487 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:26.491659 18487 raft_consensus.cc:515] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:26.493497 18487 leader_election.cc:290] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 4 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:26.494489 18697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:26.494516 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2156261889b40898ea60bd3002d1ccf" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:26.495250 18697 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:26.495355 18413 raft_consensus.cc:3055] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:26.495774 18413 raft_consensus.cc:740] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:26.496400 18413 consensus_queue.cc:260] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:26.497279 18413 raft_consensus.cc:3060] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:26.501680 18697 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 4.
I20260709 13:06:26.503737 18454 leader_election.cc:304] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:26.503785 18413 raft_consensus.cc:2468] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 4.
I20260709 13:06:26.504516 18659 raft_consensus.cc:2804] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:26.505028 18659 raft_consensus.cc:697] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 4 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:26.505875 18659 consensus_queue.cc:237] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:26.512816 18236 catalog_manager.cc:5697] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 3 to 4, leader changed from 31250b946a464780846c6ea7b435e870 (127.7.78.66) to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 4 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:26.777562 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:26.799865 18737 raft_consensus.cc:993] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870: : Instructing follower ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:26.800616 18795 raft_consensus.cc:1081] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Signalling peer ac771fb495d84241b8a5dc7056ff239f to start an election
I20260709 13:06:26.802393 18487 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
 from {username='slave'} at 127.0.0.1:47274
I20260709 13:06:26.802973 18487 raft_consensus.cc:493] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:26.803283 18487 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:26.808878 18487 raft_consensus.cc:515] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:26.810631 18487 leader_election.cc:290] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 3 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781)
I20260709 13:06:26.811740 18413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "31250b946a464780846c6ea7b435e870"
I20260709 13:06:26.811754 18698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11892abfca5043928cb0890fde5e66b3" candidate_uuid: "ac771fb495d84241b8a5dc7056ff239f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:26.812539 18413 raft_consensus.cc:3055] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:26.812697 18698 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:26.812963 18413 raft_consensus.cc:740] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:26.813964 18413 consensus_queue.cc:260] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:26.815413 18413 raft_consensus.cc:3060] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:26.821975 18698 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 3.
I20260709 13:06:26.822515 18413 raft_consensus.cc:2468] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac771fb495d84241b8a5dc7056ff239f in term 3.
I20260709 13:06:26.823559 18454 leader_election.cc:304] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa72f0c464794888866a56f660990608, ac771fb495d84241b8a5dc7056ff239f; no voters: 
I20260709 13:06:26.824262 18659 raft_consensus.cc:2804] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:26.824739 18659 raft_consensus.cc:697] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 3 LEADER]: Becoming Leader. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:26.825806 18659 consensus_queue.cc:237] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:26.836298 18236 catalog_manager.cc:5697] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f reported cstate change: term changed from 2 to 3, leader changed from 31250b946a464780846c6ea7b435e870 (127.7.78.66) to ac771fb495d84241b8a5dc7056ff239f (127.7.78.67). New cstate: current_term: 3 leader_uuid: "ac771fb495d84241b8a5dc7056ff239f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:27.000674 18698 raft_consensus.cc:1275] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 4 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:27.001961 18659 consensus_queue.cc:1048] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:27.017495 18413 raft_consensus.cc:1275] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:27.019009 18805 consensus_queue.cc:1048] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:27.272157 18698 raft_consensus.cc:1275] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:27.273662 18805 consensus_queue.cc:1048] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:27.284257 18413 raft_consensus.cc:1275] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Refusing update from remote peer ac771fb495d84241b8a5dc7056ff239f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:27.285791 18659 consensus_queue.cc:1048] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [LEADER]: Connected to new peer: Peer: permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:27.288635  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:27.298864 18487 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:27.299553 18487 raft_consensus.cc:606] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:27.303033 18487 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0"
dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
mode: GRACEFUL
new_leader_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:47290
I20260709 13:06:27.303658 18487 raft_consensus.cc:606] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Received request to transfer leadership to TS 31250b946a464780846c6ea7b435e870
I20260709 13:06:27.304997  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:27.305663  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:27.306424  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:27.401420 18659 raft_consensus.cc:993] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:27.402064 18805 raft_consensus.cc:1081] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:27.403599 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:27.404074 18413 raft_consensus.cc:493] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:27.404366 18413 raft_consensus.cc:3060] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:27.409512 18413 raft_consensus.cc:515] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:27.411064 18413 leader_election.cc:290] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:27.412415 18698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:27.412405 18487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6dee6d90d94db3bc0e31f7748252d0" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:27.413506 18698 raft_consensus.cc:3060] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:27.413565 18487 raft_consensus.cc:3055] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:27.414060 18487 raft_consensus.cc:740] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:27.415324 18487 consensus_queue.cc:260] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:27.416743 18487 raft_consensus.cc:3060] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:27.422809 18698 raft_consensus.cc:2468] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:27.423835 18697 raft_consensus.cc:1240] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Rejecting Update request from peer ac771fb495d84241b8a5dc7056ff239f for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:27.425220 18380 leader_election.cc:304] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:27.425364 18487 raft_consensus.cc:2468] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 2.
I20260709 13:06:27.426448 18795 raft_consensus.cc:2804] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:27.427651 18795 raft_consensus.cc:697] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:27.428766 18795 consensus_queue.cc:237] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:27.436887 18236 catalog_manager.cc:5697] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 1 to 2, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:27.778581 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:27.851410 18803 raft_consensus.cc:993] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: : Instructing follower 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:27.852061 18659 raft_consensus.cc:1081] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 LEADER]: Signalling peer 31250b946a464780846c6ea7b435e870 to start an election
I20260709 13:06:27.853567 18413 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a"
dest_uuid: "31250b946a464780846c6ea7b435e870"
 from {username='slave'} at 127.0.0.1:59678
I20260709 13:06:27.854054 18413 raft_consensus.cc:493] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:27.854393 18413 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:27.859405 18413 raft_consensus.cc:515] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:27.860803 18413 leader_election.cc:290] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 5 election: Requested vote from peers aa72f0c464794888866a56f660990608 (127.7.78.65:45805), ac771fb495d84241b8a5dc7056ff239f (127.7.78.67:37131)
I20260709 13:06:27.861879 18487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ac771fb495d84241b8a5dc7056ff239f"
I20260709 13:06:27.861760 18697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07bc6bf2f52c4c418867c6bbc31dbc6a" candidate_uuid: "31250b946a464780846c6ea7b435e870" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "aa72f0c464794888866a56f660990608"
I20260709 13:06:27.862592 18487 raft_consensus.cc:3055] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:06:27.862687 18697 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:27.863083 18487 raft_consensus.cc:740] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac771fb495d84241b8a5dc7056ff239f, State: Running, Role: LEADER
I20260709 13:06:27.864135 18487 consensus_queue.cc:260] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:27.864951 18698 raft_consensus.cc:1275] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:27.865453 18487 raft_consensus.cc:3060] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:27.866284 18810 consensus_queue.cc:1048] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:27.871361 18697 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 5.
I20260709 13:06:27.873128 18380 leader_election.cc:304] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31250b946a464780846c6ea7b435e870, aa72f0c464794888866a56f660990608; no voters: 
I20260709 13:06:27.873134 18487 raft_consensus.cc:2468] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31250b946a464780846c6ea7b435e870 in term 5.
I20260709 13:06:27.874545 18795 raft_consensus.cc:2804] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:06:27.875691 18795 raft_consensus.cc:697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 5 LEADER]: Becoming Leader. State: Replica: 31250b946a464780846c6ea7b435e870, State: Running, Role: LEADER
I20260709 13:06:27.876818 18795 consensus_queue.cc:237] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } }
I20260709 13:06:27.884104 18487 raft_consensus.cc:1275] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:27.885922 18810 consensus_queue.cc:1048] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:27.887605 18236 catalog_manager.cc:5697] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 reported cstate change: term changed from 4 to 5, leader changed from ac771fb495d84241b8a5dc7056ff239f (127.7.78.67) to 31250b946a464780846c6ea7b435e870 (127.7.78.66). New cstate: current_term: 5 leader_uuid: "31250b946a464780846c6ea7b435e870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31250b946a464780846c6ea7b435e870" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:28.313290  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:28.315471 18487 raft_consensus.cc:1275] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 5 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:06:28.317008 18810 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac771fb495d84241b8a5dc7056ff239f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:06:28.330828 18698 raft_consensus.cc:1275] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 5 FOLLOWER]: Refusing update from remote peer 31250b946a464780846c6ea7b435e870: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:06:28.332340 18817 consensus_queue.cc:1048] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa72f0c464794888866a56f660990608" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:06:28.352835  7481 tablet_server.cc:179] TabletServer@127.7.78.65:45805 shutting down...
I20260709 13:06:28.384065  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:28.384992  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.385890  7481 raft_consensus.cc:2243] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.386526  7481 raft_consensus.cc:2272] T 07bc6bf2f52c4c418867c6bbc31dbc6a P aa72f0c464794888866a56f660990608 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.389858  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.390492  7481 raft_consensus.cc:2243] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.391009  7481 raft_consensus.cc:2272] T 9db35f4181aa46adb426b1bae078a0df P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.393489  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.394099  7481 raft_consensus.cc:2243] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.394584  7481 raft_consensus.cc:2272] T f2156261889b40898ea60bd3002d1ccf P aa72f0c464794888866a56f660990608 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.397176  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.397985  7481 raft_consensus.cc:2243] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.398401  7481 raft_consensus.cc:2272] T 087ed505256448bfafd264d7cb18396f P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.400638  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.401125  7481 raft_consensus.cc:2243] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.401607  7481 raft_consensus.cc:2272] T 7d9d73a52af14ff69e7996ebcc71131d P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.403781  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.404325  7481 raft_consensus.cc:2243] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.404778  7481 raft_consensus.cc:2272] T ea09119560864fdabe6de1bde69b8f51 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.407172  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.407789  7481 raft_consensus.cc:2243] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.408219  7481 raft_consensus.cc:2272] T 11892abfca5043928cb0890fde5e66b3 P aa72f0c464794888866a56f660990608 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.410506  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.410998  7481 raft_consensus.cc:2243] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.411389  7481 raft_consensus.cc:2272] T 9b6dee6d90d94db3bc0e31f7748252d0 P aa72f0c464794888866a56f660990608 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.414187  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.414673  7481 raft_consensus.cc:2243] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.415048  7481 raft_consensus.cc:2272] T 104363652bb743148f6c64f81c33060e P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.417282  7481 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:28.417804  7481 raft_consensus.cc:2243] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.418212  7481 raft_consensus.cc:2272] T db070baeeb2b411fb19fed64cc792ab8 P aa72f0c464794888866a56f660990608 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:28.425204 18380 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 95 similar messages]
W20260709 13:06:28.428988 18380 consensus_peers.cc:597] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:28.453507 18454 consensus_peers.cc:597] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:28.454676  7481 tablet_server.cc:196] TabletServer@127.7.78.65:45805 shutdown complete.
W20260709 13:06:28.463699 18454 consensus_peers.cc:597] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:28.486056  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
W20260709 13:06:28.498598 18380 consensus_peers.cc:597] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:28.517632  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:28.518426  7481 tablet_replica.cc:333] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.519320  7481 raft_consensus.cc:2243] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.520165  7481 raft_consensus.cc:2272] T 9b6dee6d90d94db3bc0e31f7748252d0 P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.522940  7481 tablet_replica.cc:333] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.523574  7481 raft_consensus.cc:2243] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.524425  7481 raft_consensus.cc:2272] T db070baeeb2b411fb19fed64cc792ab8 P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.526810  7481 tablet_replica.cc:333] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.527482  7481 raft_consensus.cc:2243] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.528276  7481 raft_consensus.cc:2272] T 07bc6bf2f52c4c418867c6bbc31dbc6a P 31250b946a464780846c6ea7b435e870 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.531114  7481 tablet_replica.cc:333] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.531803  7481 raft_consensus.cc:2243] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.532333  7481 raft_consensus.cc:2272] T 104363652bb743148f6c64f81c33060e P 31250b946a464780846c6ea7b435e870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.534579  7481 tablet_replica.cc:333] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.535112  7481 raft_consensus.cc:2243] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.535972  7481 raft_consensus.cc:2272] T 9db35f4181aa46adb426b1bae078a0df P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.538273  7481 tablet_replica.cc:333] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.538829  7481 raft_consensus.cc:2243] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.539669  7481 raft_consensus.cc:2272] T 087ed505256448bfafd264d7cb18396f P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.542061  7481 tablet_replica.cc:333] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.542683  7481 raft_consensus.cc:2243] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.543483  7481 raft_consensus.cc:2272] T 7d9d73a52af14ff69e7996ebcc71131d P 31250b946a464780846c6ea7b435e870 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.546119  7481 tablet_replica.cc:333] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.546751  7481 raft_consensus.cc:2243] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.547250  7481 raft_consensus.cc:2272] T f2156261889b40898ea60bd3002d1ccf P 31250b946a464780846c6ea7b435e870 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:28.549428 18454 consensus_peers.cc:597] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:28.550141  7481 tablet_replica.cc:333] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.550813  7481 raft_consensus.cc:2243] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.551265  7481 raft_consensus.cc:2272] T 11892abfca5043928cb0890fde5e66b3 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.553738  7481 tablet_replica.cc:333] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870: stopping tablet replica
I20260709 13:06:28.554297  7481 raft_consensus.cc:2243] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.554821  7481 raft_consensus.cc:2272] T ea09119560864fdabe6de1bde69b8f51 P 31250b946a464780846c6ea7b435e870 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.589053  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
W20260709 13:06:28.603475 18453 consensus_peers.cc:597] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f -> Peer 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781): Couldn't send request to peer 31250b946a464780846c6ea7b435e870. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:37781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:28.618706  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
W20260709 13:06:28.653501 18454 consensus_peers.cc:597] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f -> Peer aa72f0c464794888866a56f660990608 (127.7.78.65:45805): Couldn't send request to peer aa72f0c464794888866a56f660990608. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:45805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:28.655371  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:28.656476  7481 tablet_replica.cc:333] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.657727  7481 raft_consensus.cc:2243] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.658533  7481 raft_consensus.cc:2272] T db070baeeb2b411fb19fed64cc792ab8 P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:28.658671 18453 consensus_peers.cc:597] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f -> Peer 31250b946a464780846c6ea7b435e870 (127.7.78.66:37781): Couldn't send request to peer 31250b946a464780846c6ea7b435e870. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:37781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:28.661517  7481 tablet_replica.cc:333] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.662302  7481 raft_consensus.cc:2243] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.662878  7481 raft_consensus.cc:2272] T 9b6dee6d90d94db3bc0e31f7748252d0 P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.665584  7481 tablet_replica.cc:333] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.666275  7481 raft_consensus.cc:2243] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.667178  7481 raft_consensus.cc:2272] T 104363652bb743148f6c64f81c33060e P ac771fb495d84241b8a5dc7056ff239f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.669735  7481 tablet_replica.cc:333] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.670361  7481 raft_consensus.cc:2243] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.670998  7481 raft_consensus.cc:2272] T 9db35f4181aa46adb426b1bae078a0df P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.673489  7481 tablet_replica.cc:333] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.674096  7481 raft_consensus.cc:2243] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.674547  7481 raft_consensus.cc:2272] T 07bc6bf2f52c4c418867c6bbc31dbc6a P ac771fb495d84241b8a5dc7056ff239f [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.677958  7481 tablet_replica.cc:333] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.678581  7481 raft_consensus.cc:2243] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.679127  7481 raft_consensus.cc:2272] T 087ed505256448bfafd264d7cb18396f P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.681352  7481 tablet_replica.cc:333] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.681980  7481 raft_consensus.cc:2243] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:28.682426  7481 raft_consensus.cc:2272] T 7d9d73a52af14ff69e7996ebcc71131d P ac771fb495d84241b8a5dc7056ff239f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.684916  7481 tablet_replica.cc:333] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.685571  7481 raft_consensus.cc:2243] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.686461  7481 raft_consensus.cc:2272] T f2156261889b40898ea60bd3002d1ccf P ac771fb495d84241b8a5dc7056ff239f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.688897  7481 tablet_replica.cc:333] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.689488  7481 raft_consensus.cc:2243] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.690409  7481 raft_consensus.cc:2272] T ea09119560864fdabe6de1bde69b8f51 P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.692728  7481 tablet_replica.cc:333] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f: stopping tablet replica
I20260709 13:06:28.693202  7481 raft_consensus.cc:2243] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.694074  7481 raft_consensus.cc:2272] T 11892abfca5043928cb0890fde5e66b3 P ac771fb495d84241b8a5dc7056ff239f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.730877  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
I20260709 13:06:28.758085  7481 master.cc:562] Master@127.7.78.126:43571 shutting down...
W20260709 13:06:28.780138 18285 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:28.787537  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:28.788117  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:28.788461  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P d0f6df7450ad403dbb99034ba2facc14: stopping tablet replica
I20260709 13:06:28.812081  7481 master.cc:584] Master@127.7.78.126:43571 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (13242 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 13:06:28.861166  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:34015
I20260709 13:06:28.862406  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:28.869095 18833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:28.869187 18832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:28.870275 18835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:28.870543  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:28.871742  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:28.871944  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:28.872076  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602388872057 us; error 0 us; skew 500 ppm
I20260709 13:06:28.872594  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:28.875762  7481 webserver.cc:533] Webserver started at http://127.7.78.126:45565/ using document root <none> and password file <none>
I20260709 13:06:28.876330  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:28.876514  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:28.876806  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:28.878145  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "f8ef2b43961b47209f0b754608ebbc0b"
format_stamp: "Formatted at 2026-07-09 13:06:28 on dist-test-slave-50cs"
I20260709 13:06:28.883198  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:06:28.887233 18840 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:28.888152  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 13:06:28.888451  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "f8ef2b43961b47209f0b754608ebbc0b"
format_stamp: "Formatted at 2026-07-09 13:06:28 on dist-test-slave-50cs"
I20260709 13:06:28.888715  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:28.912138  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:28.913475  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:28.954764  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:34015
I20260709 13:06:28.955056 18891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:34015 every 8 connection(s)
I20260709 13:06:28.958921 18892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:28.972386 18892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b: Bootstrap starting.
I20260709 13:06:28.977998 18892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:28.982880 18892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b: No bootstrap required, opened a new log
I20260709 13:06:28.985647 18892 raft_consensus.cc:359] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef2b43961b47209f0b754608ebbc0b" member_type: VOTER }
I20260709 13:06:28.986097 18892 raft_consensus.cc:385] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:28.986399 18892 raft_consensus.cc:740] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ef2b43961b47209f0b754608ebbc0b, State: Initialized, Role: FOLLOWER
I20260709 13:06:28.987149 18892 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef2b43961b47209f0b754608ebbc0b" member_type: VOTER }
I20260709 13:06:28.987675 18892 raft_consensus.cc:399] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:28.987942 18892 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:28.988303 18892 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:28.994261 18892 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef2b43961b47209f0b754608ebbc0b" member_type: VOTER }
I20260709 13:06:28.994899 18892 leader_election.cc:304] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8ef2b43961b47209f0b754608ebbc0b; no voters: 
I20260709 13:06:28.996182 18892 leader_election.cc:290] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:28.996644 18895 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:28.998098 18895 raft_consensus.cc:697] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 1 LEADER]: Becoming Leader. State: Replica: f8ef2b43961b47209f0b754608ebbc0b, State: Running, Role: LEADER
I20260709 13:06:28.998799 18895 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef2b43961b47209f0b754608ebbc0b" member_type: VOTER }
I20260709 13:06:28.999481 18892 sys_catalog.cc:565] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:29.003535 18896 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8ef2b43961b47209f0b754608ebbc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef2b43961b47209f0b754608ebbc0b" member_type: VOTER } }
I20260709 13:06:29.004410 18896 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:29.004333 18897 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ef2b43961b47209f0b754608ebbc0b. Latest consensus state: current_term: 1 leader_uuid: "f8ef2b43961b47209f0b754608ebbc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ef2b43961b47209f0b754608ebbc0b" member_type: VOTER } }
I20260709 13:06:29.005123 18897 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:29.010548 18902 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:29.015715 18902 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:29.016991  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:29.025923 18902 catalog_manager.cc:1383] Generated new cluster ID: aec4b6c0a2e8446a85ceb3d40b216f5f
I20260709 13:06:29.026327 18902 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:29.057462 18902 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:29.058874 18902 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:29.085027 18902 catalog_manager.cc:6070] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b: Generated new TSK 0
I20260709 13:06:29.085691 18902 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:29.149199  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:29.155458 18913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:29.157107 18914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:29.158603 18916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:29.159448  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:29.160475  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:29.160655  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:29.160946  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602389160929 us; error 0 us; skew 500 ppm
I20260709 13:06:29.161506  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:29.163987  7481 webserver.cc:533] Webserver started at http://127.7.78.65:41217/ using document root <none> and password file <none>
I20260709 13:06:29.164455  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:29.164635  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:29.164875  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:29.166119  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "bc714c8ec5f44959af1e9591ed9b5072"
format_stamp: "Formatted at 2026-07-09 13:06:29 on dist-test-slave-50cs"
I20260709 13:06:29.171128  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:06:29.175115 18921 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.176082  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 13:06:29.176465  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "bc714c8ec5f44959af1e9591ed9b5072"
format_stamp: "Formatted at 2026-07-09 13:06:29 on dist-test-slave-50cs"
I20260709 13:06:29.176787  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:29.192520  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:29.193966  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:29.195508  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:29.198067  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:29.198263  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.198575  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:29.198777  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.244120  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:36913
I20260709 13:06:29.244220 18983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:36913 every 8 connection(s)
I20260709 13:06:29.249799  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:29.259505 18988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:29.259510 18989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:29.264127 18984 heartbeater.cc:344] Connected to a master server at 127.7.78.126:34015
W20260709 13:06:29.264256 18991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:29.264847  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:29.264943 18984 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:29.266008 18984 heartbeater.cc:507] Master 127.7.78.126:34015 requested a full tablet report, sending...
I20260709 13:06:29.266064  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:29.266471  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:29.266667  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602389266644 us; error 0 us; skew 500 ppm
I20260709 13:06:29.267328  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:29.268330 18857 ts_manager.cc:194] Registered new tserver with Master: bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913)
I20260709 13:06:29.270311 18857 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58166
I20260709 13:06:29.270653  7481 webserver.cc:533] Webserver started at http://127.7.78.66:33733/ using document root <none> and password file <none>
I20260709 13:06:29.271324  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:29.271561  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:29.271862  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:29.273351  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
format_stamp: "Formatted at 2026-07-09 13:06:29 on dist-test-slave-50cs"
I20260709 13:06:29.278507  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:06:29.282618 18996 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.283493  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:06:29.283871  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
format_stamp: "Formatted at 2026-07-09 13:06:29 on dist-test-slave-50cs"
I20260709 13:06:29.284215  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:29.303893  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:29.305289  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:29.306977  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:29.309372  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:29.309710  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.309975  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:29.310168  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.354646  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:40923
I20260709 13:06:29.354842 19058 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:40923 every 8 connection(s)
I20260709 13:06:29.360147  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:29.368201 19062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:29.371513 19063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:29.374996  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:29.375715 19059 heartbeater.cc:344] Connected to a master server at 127.7.78.126:34015
W20260709 13:06:29.376092 19065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:29.376226 19059 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:29.377365  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:29.377718  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:29.377638 19059 heartbeater.cc:507] Master 127.7.78.126:34015 requested a full tablet report, sending...
I20260709 13:06:29.378145  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602389378121 us; error 0 us; skew 500 ppm
I20260709 13:06:29.379105  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:29.380671 18857 ts_manager.cc:194] Registered new tserver with Master: e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66:40923)
I20260709 13:06:29.381940  7481 webserver.cc:533] Webserver started at http://127.7.78.67:43021/ using document root <none> and password file <none>
I20260709 13:06:29.382403 18857 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58178
I20260709 13:06:29.382532  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:29.382915  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:29.383236  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:29.384622  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "46aacb55f19043db9c3cb8e34968f88f"
format_stamp: "Formatted at 2026-07-09 13:06:29 on dist-test-slave-50cs"
I20260709 13:06:29.389897  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 13:06:29.393776 19070 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.394645  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.002s
I20260709 13:06:29.395021  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "46aacb55f19043db9c3cb8e34968f88f"
format_stamp: "Formatted at 2026-07-09 13:06:29 on dist-test-slave-50cs"
I20260709 13:06:29.395337  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:29.425896  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:29.427277  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:29.428972  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:29.431452  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:29.431653  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.431998  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:29.432154  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:29.477070  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:44765
I20260709 13:06:29.477169 19132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:44765 every 8 connection(s)
I20260709 13:06:29.493054 19133 heartbeater.cc:344] Connected to a master server at 127.7.78.126:34015
I20260709 13:06:29.493526 19133 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:29.494359 19133 heartbeater.cc:507] Master 127.7.78.126:34015 requested a full tablet report, sending...
I20260709 13:06:29.496505 18857 ts_manager.cc:194] Registered new tserver with Master: 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:29.497144  7481 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016233502s
I20260709 13:06:29.498104 18857 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58184
I20260709 13:06:29.524412 18857 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58192:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:29.526748 18857 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:29.574294 19024 tablet_service.cc:1511] Processing CreateTablet for tablet 53d064b1afa14f3ebf520f335cd1da76 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=a60021cb05a74b40a0c175fbf76395cf]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:29.575830 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d064b1afa14f3ebf520f335cd1da76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.578567 19023 tablet_service.cc:1511] Processing CreateTablet for tablet 34ba94ff67454e2aad007a916bd55ccb (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=a60021cb05a74b40a0c175fbf76395cf]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:29.580106 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ba94ff67454e2aad007a916bd55ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.592934 19098 tablet_service.cc:1511] Processing CreateTablet for tablet 53d064b1afa14f3ebf520f335cd1da76 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=a60021cb05a74b40a0c175fbf76395cf]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:29.594473 19098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d064b1afa14f3ebf520f335cd1da76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.598089 19097 tablet_service.cc:1511] Processing CreateTablet for tablet 34ba94ff67454e2aad007a916bd55ccb (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=a60021cb05a74b40a0c175fbf76395cf]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:29.599488 19097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ba94ff67454e2aad007a916bd55ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.601570 18948 tablet_service.cc:1511] Processing CreateTablet for tablet 34ba94ff67454e2aad007a916bd55ccb (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=a60021cb05a74b40a0c175fbf76395cf]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:29.601640 18949 tablet_service.cc:1511] Processing CreateTablet for tablet 53d064b1afa14f3ebf520f335cd1da76 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=a60021cb05a74b40a0c175fbf76395cf]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:29.602998 18948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ba94ff67454e2aad007a916bd55ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.605590 18949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d064b1afa14f3ebf520f335cd1da76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.606721 19148 tablet_bootstrap.cc:492] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61: Bootstrap starting.
I20260709 13:06:29.617897 19148 tablet_bootstrap.cc:654] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.643091 19150 tablet_bootstrap.cc:492] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f: Bootstrap starting.
I20260709 13:06:29.645874 19148 tablet_bootstrap.cc:492] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61: No bootstrap required, opened a new log
I20260709 13:06:29.646503 19148 ts_tablet_manager.cc:1403] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent bootstrapping tablet: real 0.040s	user 0.016s	sys 0.015s
I20260709 13:06:29.649791 19151 tablet_bootstrap.cc:492] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072: Bootstrap starting.
I20260709 13:06:29.650264 19150 tablet_bootstrap.cc:654] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.649869 19148 raft_consensus.cc:359] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.651176 19148 raft_consensus.cc:385] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.651523 19148 raft_consensus.cc:740] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.652396 19148 consensus_queue.cc:260] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.659549 19151 tablet_bootstrap.cc:654] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.660406 19148 ts_tablet_manager.cc:1434] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.006s
I20260709 13:06:29.660677 19059 heartbeater.cc:499] Master 127.7.78.126:34015 was elected leader, sending a full tablet report...
I20260709 13:06:29.661702 19148 tablet_bootstrap.cc:492] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61: Bootstrap starting.
I20260709 13:06:29.662580 19150 tablet_bootstrap.cc:492] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f: No bootstrap required, opened a new log
I20260709 13:06:29.663108 19150 ts_tablet_manager.cc:1403] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260709 13:06:29.666244 19151 tablet_bootstrap.cc:492] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072: No bootstrap required, opened a new log
I20260709 13:06:29.666868 19151 ts_tablet_manager.cc:1403] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.005s
I20260709 13:06:29.666486 19150 raft_consensus.cc:359] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.667321 19150 raft_consensus.cc:385] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.667659 19150 raft_consensus.cc:740] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46aacb55f19043db9c3cb8e34968f88f, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.668447 19150 consensus_queue.cc:260] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.669127 19148 tablet_bootstrap.cc:654] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.671779 19151 raft_consensus.cc:359] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.672875 19151 raft_consensus.cc:385] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.673271 19151 raft_consensus.cc:740] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.675379 19151 consensus_queue.cc:260] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.688875 19133 heartbeater.cc:499] Master 127.7.78.126:34015 was elected leader, sending a full tablet report...
I20260709 13:06:29.688946 18984 heartbeater.cc:499] Master 127.7.78.126:34015 was elected leader, sending a full tablet report...
I20260709 13:06:29.689255 19151 ts_tablet_manager.cc:1434] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.013s
I20260709 13:06:29.689591 19150 ts_tablet_manager.cc:1434] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f: Time spent starting tablet: real 0.026s	user 0.020s	sys 0.008s
I20260709 13:06:29.691624 19150 tablet_bootstrap.cc:492] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f: Bootstrap starting.
I20260709 13:06:29.693300 19148 tablet_bootstrap.cc:492] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61: No bootstrap required, opened a new log
I20260709 13:06:29.693902 19148 ts_tablet_manager.cc:1403] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.017s
I20260709 13:06:29.694255 19151 tablet_bootstrap.cc:492] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072: Bootstrap starting.
I20260709 13:06:29.697124 19148 raft_consensus.cc:359] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.698014 19148 raft_consensus.cc:385] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.698324 19148 raft_consensus.cc:740] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.698525 19150 tablet_bootstrap.cc:654] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.699268 19148 consensus_queue.cc:260] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.702350 19151 tablet_bootstrap.cc:654] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.711498 19148 ts_tablet_manager.cc:1434] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent starting tablet: real 0.017s	user 0.002s	sys 0.004s
I20260709 13:06:29.719120 19151 tablet_bootstrap.cc:492] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072: No bootstrap required, opened a new log
I20260709 13:06:29.719466 19150 tablet_bootstrap.cc:492] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f: No bootstrap required, opened a new log
I20260709 13:06:29.719671 19151 ts_tablet_manager.cc:1403] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072: Time spent bootstrapping tablet: real 0.026s	user 0.007s	sys 0.014s
I20260709 13:06:29.720041 19150 ts_tablet_manager.cc:1403] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.015s
I20260709 13:06:29.722234 19151 raft_consensus.cc:359] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.722494 19150 raft_consensus.cc:359] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.723192 19151 raft_consensus.cc:385] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.723424 19150 raft_consensus.cc:385] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.723652 19151 raft_consensus.cc:740] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.723969 19150 raft_consensus.cc:740] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46aacb55f19043db9c3cb8e34968f88f, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.724534 19151 consensus_queue.cc:260] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.724637 19150 consensus_queue.cc:260] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.726876 19151 ts_tablet_manager.cc:1434] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:06:29.726927 19150 ts_tablet_manager.cc:1434] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:06:29.781081 19155 raft_consensus.cc:493] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:29.781754 19155 raft_consensus.cc:515] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.783878 19155 leader_election.cc:290] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66:40923), bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913)
I20260709 13:06:29.794338 19034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76" candidate_uuid: "46aacb55f19043db9c3cb8e34968f88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" is_pre_election: true
I20260709 13:06:29.795250 19034 raft_consensus.cc:2468] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46aacb55f19043db9c3cb8e34968f88f in term 0.
I20260709 13:06:29.795197 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76" candidate_uuid: "46aacb55f19043db9c3cb8e34968f88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072" is_pre_election: true
I20260709 13:06:29.796176 18959 raft_consensus.cc:2468] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46aacb55f19043db9c3cb8e34968f88f in term 0.
I20260709 13:06:29.796959 19074 leader_election.cc:304] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:29.797816 19155 raft_consensus.cc:2804] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:29.798168 19155 raft_consensus.cc:493] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:29.798445 19155 raft_consensus.cc:3060] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:29.804033 19155 raft_consensus.cc:515] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.805764 19155 leader_election.cc:290] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [CANDIDATE]: Term 1 election: Requested vote from peers e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66:40923), bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913)
I20260709 13:06:29.806725 19034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76" candidate_uuid: "46aacb55f19043db9c3cb8e34968f88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
I20260709 13:06:29.806871 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76" candidate_uuid: "46aacb55f19043db9c3cb8e34968f88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
I20260709 13:06:29.807332 19034 raft_consensus.cc:3060] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:29.807436 18959 raft_consensus.cc:3060] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:29.813179 19034 raft_consensus.cc:2468] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46aacb55f19043db9c3cb8e34968f88f in term 1.
I20260709 13:06:29.813153 18959 raft_consensus.cc:2468] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46aacb55f19043db9c3cb8e34968f88f in term 1.
I20260709 13:06:29.814601 19074 leader_election.cc:304] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:29.815328 19155 raft_consensus.cc:2804] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:29.816272 19155 raft_consensus.cc:697] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 LEADER]: Becoming Leader. State: Replica: 46aacb55f19043db9c3cb8e34968f88f, State: Running, Role: LEADER
I20260709 13:06:29.817070 19155 consensus_queue.cc:237] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.818138 19152 raft_consensus.cc:493] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:29.818596 19152 raft_consensus.cc:515] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.820899 19152 leader_election.cc:290] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:29.826114 18857 catalog_manager.cc:5697] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f reported cstate change: term changed from 0 to 1, leader changed from <none> to 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67). New cstate: current_term: 1 leader_uuid: "46aacb55f19043db9c3cb8e34968f88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:29.832115 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072" is_pre_election: true
I20260709 13:06:29.832806 18959 raft_consensus.cc:2468] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 0.
I20260709 13:06:29.834023 18998 leader_election.cc:304] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc714c8ec5f44959af1e9591ed9b5072, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:29.834860 19152 raft_consensus.cc:2804] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:29.835211 19152 raft_consensus.cc:493] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:29.835510 19152 raft_consensus.cc:3060] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:29.836112 19108 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aacb55f19043db9c3cb8e34968f88f" is_pre_election: true
I20260709 13:06:29.836776 19108 raft_consensus.cc:2468] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 0.
I20260709 13:06:29.841856 19152 raft_consensus.cc:515] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.843544 19152 leader_election.cc:290] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 election: Requested vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:29.844544 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
I20260709 13:06:29.844576 19108 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
I20260709 13:06:29.845254 18959 raft_consensus.cc:3060] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:29.845357 19108 raft_consensus.cc:3060] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:29.850379 18959 raft_consensus.cc:2468] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 1.
I20260709 13:06:29.850880 19108 raft_consensus.cc:2468] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 1.
I20260709 13:06:29.851750 18998 leader_election.cc:304] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc714c8ec5f44959af1e9591ed9b5072, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:29.852798 19152 raft_consensus.cc:2804] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:29.854137 19152 raft_consensus.cc:697] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Becoming Leader. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Running, Role: LEADER
I20260709 13:06:29.854993 19152 consensus_queue.cc:237] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.865713 18857 catalog_manager.cc:5697] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:29.917296 18857 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58206:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:29.919450 18857 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:29.943011 18949 tablet_service.cc:1511] Processing CreateTablet for tablet d2c052597b67405e924cd85f58127704 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=d3dc680b08c446fc830e7ef75151447b]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:29.944348 18949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c052597b67405e924cd85f58127704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.944465 19023 tablet_service.cc:1511] Processing CreateTablet for tablet d2c052597b67405e924cd85f58127704 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=d3dc680b08c446fc830e7ef75151447b]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:29.945834 19023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c052597b67405e924cd85f58127704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.946419 19097 tablet_service.cc:1511] Processing CreateTablet for tablet d2c052597b67405e924cd85f58127704 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=d3dc680b08c446fc830e7ef75151447b]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:29.947705 19097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c052597b67405e924cd85f58127704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.948086 18948 tablet_service.cc:1511] Processing CreateTablet for tablet 7227b27286d84417b7e77abfca0a1781 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=d3dc680b08c446fc830e7ef75151447b]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:29.949384 18948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7227b27286d84417b7e77abfca0a1781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.960875 19024 tablet_service.cc:1511] Processing CreateTablet for tablet 7227b27286d84417b7e77abfca0a1781 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=d3dc680b08c446fc830e7ef75151447b]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:29.962356 19024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7227b27286d84417b7e77abfca0a1781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.963413 19148 tablet_bootstrap.cc:492] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61: Bootstrap starting.
I20260709 13:06:29.970222 19148 tablet_bootstrap.cc:654] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.970369 19151 tablet_bootstrap.cc:492] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072: Bootstrap starting.
I20260709 13:06:29.975658 19150 tablet_bootstrap.cc:492] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f: Bootstrap starting.
I20260709 13:06:29.977035 19151 tablet_bootstrap.cc:654] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.984086 19150 tablet_bootstrap.cc:654] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.985081 19148 tablet_bootstrap.cc:492] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61: No bootstrap required, opened a new log
I20260709 13:06:29.985733 19148 ts_tablet_manager.cc:1403] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.005s
I20260709 13:06:29.986258 19151 tablet_bootstrap.cc:492] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072: No bootstrap required, opened a new log
I20260709 13:06:29.985935 19097 tablet_service.cc:1511] Processing CreateTablet for tablet 7227b27286d84417b7e77abfca0a1781 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=d3dc680b08c446fc830e7ef75151447b]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:29.986820 19151 ts_tablet_manager.cc:1403] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.001s
I20260709 13:06:29.987879 19097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7227b27286d84417b7e77abfca0a1781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:29.990089 19151 raft_consensus.cc:359] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.991052 19151 raft_consensus.cc:385] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.991375 19151 raft_consensus.cc:740] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.989038 19148 raft_consensus.cc:359] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.992172 19151 consensus_queue.cc:260] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.992669 19148 raft_consensus.cc:385] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:29.992998 19148 raft_consensus.cc:740] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Initialized, Role: FOLLOWER
I20260709 13:06:29.993849 19148 consensus_queue.cc:260] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:29.996492 19151 ts_tablet_manager.cc:1434] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:29.996544 19150 tablet_bootstrap.cc:492] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f: No bootstrap required, opened a new log
I20260709 13:06:29.997898 19150 ts_tablet_manager.cc:1403] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f: Time spent bootstrapping tablet: real 0.023s	user 0.017s	sys 0.003s
I20260709 13:06:30.001463 19150 raft_consensus.cc:359] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.002002 19151 tablet_bootstrap.cc:492] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072: Bootstrap starting.
I20260709 13:06:30.002388 19150 raft_consensus.cc:385] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:30.002928 19150 raft_consensus.cc:740] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46aacb55f19043db9c3cb8e34968f88f, State: Initialized, Role: FOLLOWER
I20260709 13:06:30.003991 19150 consensus_queue.cc:260] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.008654 19150 ts_tablet_manager.cc:1434] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.004s
I20260709 13:06:30.010154 19151 tablet_bootstrap.cc:654] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:29.996492 19148 ts_tablet_manager.cc:1434] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:06:30.015558 19148 tablet_bootstrap.cc:492] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: Bootstrap starting.
I20260709 13:06:30.016726 19150 tablet_bootstrap.cc:492] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f: Bootstrap starting.
I20260709 13:06:30.028781 19148 tablet_bootstrap.cc:654] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:30.029944 18905 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:30.033456 19150 tablet_bootstrap.cc:654] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:30.041539 19151 tablet_bootstrap.cc:492] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072: No bootstrap required, opened a new log
I20260709 13:06:30.042003 19150 tablet_bootstrap.cc:492] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f: No bootstrap required, opened a new log
I20260709 13:06:30.041543 19148 tablet_bootstrap.cc:492] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: No bootstrap required, opened a new log
I20260709 13:06:30.042234 19151 ts_tablet_manager.cc:1403] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072: Time spent bootstrapping tablet: real 0.040s	user 0.023s	sys 0.015s
I20260709 13:06:30.042600 19150 ts_tablet_manager.cc:1403] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f: Time spent bootstrapping tablet: real 0.026s	user 0.006s	sys 0.008s
I20260709 13:06:30.042758 19148 ts_tablet_manager.cc:1403] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.010s
I20260709 13:06:30.045621 19148 raft_consensus.cc:359] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.046205 19148 raft_consensus.cc:385] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:30.046121 19150 raft_consensus.cc:359] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.046778 19148 raft_consensus.cc:740] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Initialized, Role: FOLLOWER
I20260709 13:06:30.046356 19151 raft_consensus.cc:359] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.047113 19150 raft_consensus.cc:385] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:30.047331 19151 raft_consensus.cc:385] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:30.047556 19150 raft_consensus.cc:740] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46aacb55f19043db9c3cb8e34968f88f, State: Initialized, Role: FOLLOWER
I20260709 13:06:30.047760 19151 raft_consensus.cc:740] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Initialized, Role: FOLLOWER
I20260709 13:06:30.047854 19148 consensus_queue.cc:260] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.048480 19150 consensus_queue.cc:260] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.048708 19151 consensus_queue.cc:260] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.051465 19150 ts_tablet_manager.cc:1434] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:30.052424 19151 ts_tablet_manager.cc:1434] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
I20260709 13:06:30.050468 19148 ts_tablet_manager.cc:1434] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:06:30.096539 19152 raft_consensus.cc:493] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:30.097218 19152 raft_consensus.cc:515] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.099056 19152 leader_election.cc:290] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:30.100013 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c052597b67405e924cd85f58127704" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072" is_pre_election: true
I20260709 13:06:30.100113 19108 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c052597b67405e924cd85f58127704" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aacb55f19043db9c3cb8e34968f88f" is_pre_election: true
I20260709 13:06:30.100677 18959 raft_consensus.cc:2468] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 0.
I20260709 13:06:30.100847 19108 raft_consensus.cc:2468] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 0.
I20260709 13:06:30.102003 18997 leader_election.cc:304] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:30.102905 19152 raft_consensus.cc:2804] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:30.103253 19152 raft_consensus.cc:493] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:30.103542 19152 raft_consensus.cc:3060] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:30.108857 19152 raft_consensus.cc:515] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.110646 19152 leader_election.cc:290] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 election: Requested vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:30.111670 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c052597b67405e924cd85f58127704" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
I20260709 13:06:30.111863 19108 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c052597b67405e924cd85f58127704" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
I20260709 13:06:30.112323 18959 raft_consensus.cc:3060] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:30.112521 19108 raft_consensus.cc:3060] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:30.118682 19108 raft_consensus.cc:2468] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 1.
I20260709 13:06:30.118850 18959 raft_consensus.cc:2468] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 1.
I20260709 13:06:30.119800 18997 leader_election.cc:304] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:30.120877 19152 raft_consensus.cc:2804] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:30.121472 19152 raft_consensus.cc:697] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Becoming Leader. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Running, Role: LEADER
I20260709 13:06:30.122249 19152 consensus_queue.cc:237] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.131248 18856 catalog_manager.cc:5697] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:30.215288 19155 consensus_queue.cc:1048] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:30.228657 19155 consensus_queue.cc:1048] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:30.248011 19152 raft_consensus.cc:493] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:30.248582 19152 raft_consensus.cc:515] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.250437 19152 leader_election.cc:290] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:30.251226 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072" is_pre_election: true
I20260709 13:06:30.251446 19108 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aacb55f19043db9c3cb8e34968f88f" is_pre_election: true
I20260709 13:06:30.251888 18959 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 0.
I20260709 13:06:30.252251 19108 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 0.
I20260709 13:06:30.252995 18998 leader_election.cc:304] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc714c8ec5f44959af1e9591ed9b5072, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:30.253813 19152 raft_consensus.cc:2804] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:30.254129 19152 raft_consensus.cc:493] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:30.254427 19152 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:30.259441 19152 raft_consensus.cc:515] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.261070 19152 leader_election.cc:290] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 election: Requested vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:30.262156 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
I20260709 13:06:30.262152 19108 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
I20260709 13:06:30.262835 18959 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:30.262955 19108 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:30.267266 19152 consensus_queue.cc:1048] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:30.269610 18959 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 1.
I20260709 13:06:30.269656 19108 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 1.
I20260709 13:06:30.270933 18998 leader_election.cc:304] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc714c8ec5f44959af1e9591ed9b5072, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:30.271723 19152 raft_consensus.cc:2804] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:30.272311 19152 raft_consensus.cc:697] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Becoming Leader. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Running, Role: LEADER
I20260709 13:06:30.273874 19152 consensus_queue.cc:237] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.281927 19164 consensus_queue.cc:1048] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:30.292702 18854 catalog_manager.cc:5697] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:30.384342 19107 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76"
dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
mode: GRACEFUL
new_leader_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
 from {username='slave'} at 127.0.0.1:45732
I20260709 13:06:30.384855 19107 raft_consensus.cc:606] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 LEADER]: Received request to transfer leadership to TS bc714c8ec5f44959af1e9591ed9b5072
I20260709 13:06:30.399881 19034 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb"
dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
mode: GRACEFUL
new_leader_uuid: "46aacb55f19043db9c3cb8e34968f88f"
 from {username='slave'} at 127.0.0.1:46998
I20260709 13:06:30.400426 19034 raft_consensus.cc:606] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Received request to transfer leadership to TS 46aacb55f19043db9c3cb8e34968f88f
I20260709 13:06:30.406579 19034 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2c052597b67405e924cd85f58127704"
dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
mode: GRACEFUL
new_leader_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
 from {username='slave'} at 127.0.0.1:46998
I20260709 13:06:30.407085 19034 raft_consensus.cc:606] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Received request to transfer leadership to TS bc714c8ec5f44959af1e9591ed9b5072
I20260709 13:06:30.413357 19034 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781"
dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
mode: GRACEFUL
new_leader_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
 from {username='slave'} at 127.0.0.1:46998
I20260709 13:06:30.413918 19034 raft_consensus.cc:606] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Received request to transfer leadership to TS bc714c8ec5f44959af1e9591ed9b5072
I20260709 13:06:30.638588 19161 raft_consensus.cc:993] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f: : Instructing follower bc714c8ec5f44959af1e9591ed9b5072 to start an election
I20260709 13:06:30.639276 19155 raft_consensus.cc:1081] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 LEADER]: Signalling peer bc714c8ec5f44959af1e9591ed9b5072 to start an election
I20260709 13:06:30.641093 18958 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76"
dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
 from {username='slave'} at 127.0.0.1:38282
I20260709 13:06:30.641772 18958 raft_consensus.cc:493] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:30.642197 18958 raft_consensus.cc:3060] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.648464 18958 raft_consensus.cc:515] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.653162 18958 leader_election.cc:290] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [CANDIDATE]: Term 2 election: Requested vote from peers e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66:40923), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:30.661655 19034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76" candidate_uuid: "bc714c8ec5f44959af1e9591ed9b5072" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
I20260709 13:06:30.662240 19034 raft_consensus.cc:3060] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.662810 19107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53d064b1afa14f3ebf520f335cd1da76" candidate_uuid: "bc714c8ec5f44959af1e9591ed9b5072" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
I20260709 13:06:30.663512 19107 raft_consensus.cc:3055] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:30.663868 19107 raft_consensus.cc:740] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46aacb55f19043db9c3cb8e34968f88f, State: Running, Role: LEADER
I20260709 13:06:30.664759 19107 consensus_queue.cc:260] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.666112 19107 raft_consensus.cc:3060] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.669919 19034 raft_consensus.cc:2468] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc714c8ec5f44959af1e9591ed9b5072 in term 2.
I20260709 13:06:30.671272 18925 leader_election.cc:304] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc714c8ec5f44959af1e9591ed9b5072, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:30.672317 19157 raft_consensus.cc:2804] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:30.673285 19107 raft_consensus.cc:2468] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc714c8ec5f44959af1e9591ed9b5072 in term 2.
I20260709 13:06:30.673662 19157 raft_consensus.cc:697] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Becoming Leader. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Running, Role: LEADER
I20260709 13:06:30.674852 19157 consensus_queue.cc:237] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.681807 18856 catalog_manager.cc:5697] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 reported cstate change: term changed from 1 to 2, leader changed from 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67) to bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "bc714c8ec5f44959af1e9591ed9b5072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:30.728046 19184 consensus_queue.cc:1048] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:30.732817 19186 consensus_queue.cc:1048] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:30.735692 19184 raft_consensus.cc:993] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61: : Instructing follower bc714c8ec5f44959af1e9591ed9b5072 to start an election
I20260709 13:06:30.736169 19184 raft_consensus.cc:1081] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Signalling peer bc714c8ec5f44959af1e9591ed9b5072 to start an election
I20260709 13:06:30.738874 18959 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d2c052597b67405e924cd85f58127704"
dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
 from {username='slave'} at 127.0.0.1:38296
I20260709 13:06:30.739511 18959 raft_consensus.cc:493] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:30.739915 18959 raft_consensus.cc:3060] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.747277 19186 consensus_queue.cc:1048] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:30.747203 18959 raft_consensus.cc:515] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.749178 18958 raft_consensus.cc:1240] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Rejecting Update request from peer e85ad75457cf48ee8f3f9e9fb377fa61 for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:30.750542 19186 consensus_queue.cc:1059] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:30.751580 19186 raft_consensus.cc:3055] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:30.751928 19186 raft_consensus.cc:740] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Running, Role: LEADER
I20260709 13:06:30.752980 18959 leader_election.cc:290] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [CANDIDATE]: Term 2 election: Requested vote from peers e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66:40923), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:30.752758 19186 consensus_queue.cc:260] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.753710 19106 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c052597b67405e924cd85f58127704" candidate_uuid: "bc714c8ec5f44959af1e9591ed9b5072" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
I20260709 13:06:30.754562 19034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c052597b67405e924cd85f58127704" candidate_uuid: "bc714c8ec5f44959af1e9591ed9b5072" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
I20260709 13:06:30.754621 19106 raft_consensus.cc:3060] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.753751 19186 raft_consensus.cc:3060] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.766227 19106 raft_consensus.cc:2468] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc714c8ec5f44959af1e9591ed9b5072 in term 2.
I20260709 13:06:30.767520 18922 leader_election.cc:304] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, bc714c8ec5f44959af1e9591ed9b5072; no voters: 
I20260709 13:06:30.768441 19157 raft_consensus.cc:2804] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:30.768939 19157 raft_consensus.cc:697] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Becoming Leader. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Running, Role: LEADER
I20260709 13:06:30.769827 19157 consensus_queue.cc:237] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.770087 19184 consensus_queue.cc:1048] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:30.776953 19034 raft_consensus.cc:2468] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc714c8ec5f44959af1e9591ed9b5072 in term 2.
I20260709 13:06:30.783982 19186 raft_consensus.cc:993] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: : Instructing follower bc714c8ec5f44959af1e9591ed9b5072 to start an election
I20260709 13:06:30.784466 19184 raft_consensus.cc:1081] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Signalling peer bc714c8ec5f44959af1e9591ed9b5072 to start an election
I20260709 13:06:30.787052 18959 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781"
dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
 from {username='slave'} at 127.0.0.1:38296
I20260709 13:06:30.787696 18959 raft_consensus.cc:493] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:30.788264 18959 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.787504 18856 catalog_manager.cc:5697] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 reported cstate change: term changed from 1 to 2, leader changed from e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66) to bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "bc714c8ec5f44959af1e9591ed9b5072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:30.794910 18959 raft_consensus.cc:515] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.797888 19034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "bc714c8ec5f44959af1e9591ed9b5072" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
I20260709 13:06:30.798210 19107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "bc714c8ec5f44959af1e9591ed9b5072" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
I20260709 13:06:30.798566 19034 raft_consensus.cc:3055] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:30.798801 19107 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.798949 19034 raft_consensus.cc:740] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Running, Role: LEADER
I20260709 13:06:30.797622 18959 leader_election.cc:290] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [CANDIDATE]: Term 2 election: Requested vote from peers e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66:40923), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:30.799842 19034 consensus_queue.cc:260] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.801223 19034 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.806217 19107 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc714c8ec5f44959af1e9591ed9b5072 in term 2.
I20260709 13:06:30.807413 18922 leader_election.cc:304] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, bc714c8ec5f44959af1e9591ed9b5072; no voters: 
I20260709 13:06:30.808143 19034 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc714c8ec5f44959af1e9591ed9b5072 in term 2.
I20260709 13:06:30.808372 19197 raft_consensus.cc:2804] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:30.808856 19197 raft_consensus.cc:697] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Becoming Leader. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Running, Role: LEADER
I20260709 13:06:30.809746 19197 consensus_queue.cc:237] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.816009 18856 catalog_manager.cc:5697] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 reported cstate change: term changed from 1 to 2, leader changed from e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66) to bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "bc714c8ec5f44959af1e9591ed9b5072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:30.826821 19186 raft_consensus.cc:993] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61: : Instructing follower 46aacb55f19043db9c3cb8e34968f88f to start an election
I20260709 13:06:30.827332 19184 raft_consensus.cc:1081] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Signalling peer 46aacb55f19043db9c3cb8e34968f88f to start an election
I20260709 13:06:30.828896 19107 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb"
dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
 from {username='slave'} at 127.0.0.1:45720
I20260709 13:06:30.829442 19107 raft_consensus.cc:493] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:30.829756 19107 raft_consensus.cc:3060] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.834797 19107 raft_consensus.cc:515] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.836597 19107 leader_election.cc:290] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [CANDIDATE]: Term 2 election: Requested vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66:40923)
I20260709 13:06:30.837507 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb" candidate_uuid: "46aacb55f19043db9c3cb8e34968f88f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
I20260709 13:06:30.837849 19034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ba94ff67454e2aad007a916bd55ccb" candidate_uuid: "46aacb55f19043db9c3cb8e34968f88f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
I20260709 13:06:30.838165 18959 raft_consensus.cc:3060] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.838470 19034 raft_consensus.cc:3055] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:30.838779 19034 raft_consensus.cc:740] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Running, Role: LEADER
I20260709 13:06:30.839479 19034 consensus_queue.cc:260] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.840512 19034 raft_consensus.cc:3060] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:30.843679 18959 raft_consensus.cc:2468] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46aacb55f19043db9c3cb8e34968f88f in term 2.
I20260709 13:06:30.845163 19072 leader_election.cc:304] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, bc714c8ec5f44959af1e9591ed9b5072; no voters: 
I20260709 13:06:30.846009 19155 raft_consensus.cc:2804] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:30.846052 19034 raft_consensus.cc:2468] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46aacb55f19043db9c3cb8e34968f88f in term 2.
I20260709 13:06:30.846590 19155 raft_consensus.cc:697] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 2 LEADER]: Becoming Leader. State: Replica: 46aacb55f19043db9c3cb8e34968f88f, State: Running, Role: LEADER
I20260709 13:06:30.847477 19155 consensus_queue.cc:237] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:30.853550 18856 catalog_manager.cc:5697] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f reported cstate change: term changed from 1 to 2, leader changed from e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66) to 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67). New cstate: current_term: 2 leader_uuid: "46aacb55f19043db9c3cb8e34968f88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:30.880779  7481 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 13:06:30.882478  7481 auto_leader_rebalancer.cc:452] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 13:06:30.882917  7481 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 13:06:30.897544 18959 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781"
dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
mode: GRACEFUL
new_leader_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
 from {username='slave'} at 127.0.0.1:38302
I20260709 13:06:30.898120 18959 raft_consensus.cc:606] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Received request to transfer leadership to TS e85ad75457cf48ee8f3f9e9fb377fa61
I20260709 13:06:30.899545  7481 auto_leader_rebalancer.cc:452] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 13:06:31.055207 19034 raft_consensus.cc:1275] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Refusing update from remote peer bc714c8ec5f44959af1e9591ed9b5072: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.056613 19197 consensus_queue.cc:1048] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:31.065822 19107 raft_consensus.cc:1275] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Refusing update from remote peer bc714c8ec5f44959af1e9591ed9b5072: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.067497 19193 consensus_queue.cc:1048] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:31.157542 19034 raft_consensus.cc:1275] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Refusing update from remote peer bc714c8ec5f44959af1e9591ed9b5072: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.158990 19193 consensus_queue.cc:1048] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:31.169504 19107 raft_consensus.cc:1275] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Refusing update from remote peer bc714c8ec5f44959af1e9591ed9b5072: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.171482 19201 consensus_queue.cc:1048] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:31.329782 18959 raft_consensus.cc:1275] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Refusing update from remote peer 46aacb55f19043db9c3cb8e34968f88f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.331301 19155 consensus_queue.cc:1048] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:31.341143 19034 raft_consensus.cc:1275] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Refusing update from remote peer 46aacb55f19043db9c3cb8e34968f88f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.343055 19203 consensus_queue.cc:1048] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:31.399209 19034 raft_consensus.cc:1275] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Refusing update from remote peer bc714c8ec5f44959af1e9591ed9b5072: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.400645 19201 consensus_queue.cc:1048] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:31.406261 19157 raft_consensus.cc:993] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072: : Instructing follower e85ad75457cf48ee8f3f9e9fb377fa61 to start an election
I20260709 13:06:31.406752 19193 raft_consensus.cc:1081] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Signalling peer e85ad75457cf48ee8f3f9e9fb377fa61 to start an election
I20260709 13:06:31.408692 19033 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781"
dest_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61"
 from {username='slave'} at 127.0.0.1:47004
I20260709 13:06:31.410148 19033 raft_consensus.cc:493] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:31.410570 19033 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:31.410604 19107 raft_consensus.cc:1275] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Refusing update from remote peer bc714c8ec5f44959af1e9591ed9b5072: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:31.412847 19201 consensus_queue.cc:1048] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:31.419324 19033 raft_consensus.cc:515] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:31.422538 18959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bc714c8ec5f44959af1e9591ed9b5072"
I20260709 13:06:31.422621 19107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7227b27286d84417b7e77abfca0a1781" candidate_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "46aacb55f19043db9c3cb8e34968f88f"
I20260709 13:06:31.423296 18959 raft_consensus.cc:3055] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:31.423362 19107 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:31.423898 18959 raft_consensus.cc:740] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bc714c8ec5f44959af1e9591ed9b5072, State: Running, Role: LEADER
I20260709 13:06:31.425043 18959 consensus_queue.cc:260] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:31.426308 18959 raft_consensus.cc:3060] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:31.429541 19033 leader_election.cc:290] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 3 election: Requested vote from peers bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65:36913), 46aacb55f19043db9c3cb8e34968f88f (127.7.78.67:44765)
I20260709 13:06:31.429802 19107 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 3.
I20260709 13:06:31.431288 18997 leader_election.cc:304] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46aacb55f19043db9c3cb8e34968f88f, e85ad75457cf48ee8f3f9e9fb377fa61; no voters: 
I20260709 13:06:31.432058 19184 raft_consensus.cc:2804] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:31.433213 19184 raft_consensus.cc:697] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 3 LEADER]: Becoming Leader. State: Replica: e85ad75457cf48ee8f3f9e9fb377fa61, State: Running, Role: LEADER
I20260709 13:06:31.433329 18959 raft_consensus.cc:2468] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85ad75457cf48ee8f3f9e9fb377fa61 in term 3.
I20260709 13:06:31.434134 19184 consensus_queue.cc:237] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } }
I20260709 13:06:31.440640 18854 catalog_manager.cc:5697] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 reported cstate change: term changed from 2 to 3, leader changed from bc714c8ec5f44959af1e9591ed9b5072 (127.7.78.65) to e85ad75457cf48ee8f3f9e9fb377fa61 (127.7.78.66). New cstate: current_term: 3 leader_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc714c8ec5f44959af1e9591ed9b5072" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85ad75457cf48ee8f3f9e9fb377fa61" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 40923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46aacb55f19043db9c3cb8e34968f88f" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:31.522378  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
I20260709 13:06:31.545969  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:31.546715  7481 tablet_replica.cc:333] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072: stopping tablet replica
I20260709 13:06:31.547397  7481 raft_consensus.cc:2243] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.547936  7481 raft_consensus.cc:2272] T 7227b27286d84417b7e77abfca0a1781 P bc714c8ec5f44959af1e9591ed9b5072 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.550465  7481 tablet_replica.cc:333] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072: stopping tablet replica
I20260709 13:06:31.551070  7481 raft_consensus.cc:2243] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:31.551927  7481 raft_consensus.cc:2272] T d2c052597b67405e924cd85f58127704 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.554919  7481 tablet_replica.cc:333] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072: stopping tablet replica
I20260709 13:06:31.555496  7481 raft_consensus.cc:2243] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:31.556360  7481 raft_consensus.cc:2272] T 53d064b1afa14f3ebf520f335cd1da76 P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.558815  7481 tablet_replica.cc:333] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072: stopping tablet replica
I20260709 13:06:31.559350  7481 raft_consensus.cc:2243] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.559803  7481 raft_consensus.cc:2272] T 34ba94ff67454e2aad007a916bd55ccb P bc714c8ec5f44959af1e9591ed9b5072 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.584476  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
I20260709 13:06:31.599010  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
I20260709 13:06:31.618088  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:31.618834  7481 tablet_replica.cc:333] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: stopping tablet replica
I20260709 13:06:31.619443  7481 raft_consensus.cc:2243] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:31.620172  7481 pending_rounds.cc:70] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: Trying to abort 1 pending ops.
I20260709 13:06:31.620373  7481 pending_rounds.cc:77] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7305635395322171392 op_type: NO_OP noop_request { }
I20260709 13:06:31.620676  7481 raft_consensus.cc:2889] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:06:31.621011  7481 raft_consensus.cc:2272] T 7227b27286d84417b7e77abfca0a1781 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.623529  7481 tablet_replica.cc:333] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61: stopping tablet replica
I20260709 13:06:31.624116  7481 raft_consensus.cc:2243] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.624600  7481 raft_consensus.cc:2272] T d2c052597b67405e924cd85f58127704 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.626927  7481 tablet_replica.cc:333] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61: stopping tablet replica
I20260709 13:06:31.627478  7481 raft_consensus.cc:2243] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.627925  7481 raft_consensus.cc:2272] T 53d064b1afa14f3ebf520f335cd1da76 P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.630694  7481 tablet_replica.cc:333] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61: stopping tablet replica
I20260709 13:06:31.631292  7481 raft_consensus.cc:2243] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.631824  7481 raft_consensus.cc:2272] T 34ba94ff67454e2aad007a916bd55ccb P e85ad75457cf48ee8f3f9e9fb377fa61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.647723  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
I20260709 13:06:31.661294  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
I20260709 13:06:31.685045  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:31.686151  7481 tablet_replica.cc:333] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f: stopping tablet replica
I20260709 13:06:31.686878  7481 raft_consensus.cc:2243] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.687343  7481 raft_consensus.cc:2272] T 7227b27286d84417b7e77abfca0a1781 P 46aacb55f19043db9c3cb8e34968f88f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.689978  7481 tablet_replica.cc:333] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f: stopping tablet replica
I20260709 13:06:31.690614  7481 raft_consensus.cc:2243] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.691135  7481 raft_consensus.cc:2272] T d2c052597b67405e924cd85f58127704 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.693743  7481 tablet_replica.cc:333] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f: stopping tablet replica
I20260709 13:06:31.694381  7481 raft_consensus.cc:2243] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:31.695230  7481 raft_consensus.cc:2272] T 34ba94ff67454e2aad007a916bd55ccb P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.697896  7481 tablet_replica.cc:333] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f: stopping tablet replica
I20260709 13:06:31.698415  7481 raft_consensus.cc:2243] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:31.698884  7481 raft_consensus.cc:2272] T 53d064b1afa14f3ebf520f335cd1da76 P 46aacb55f19043db9c3cb8e34968f88f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.713039  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
I20260709 13:06:31.728399  7481 master.cc:562] Master@127.7.78.126:34015 shutting down...
I20260709 13:06:31.750166  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:31.750846  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:31.751240  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P f8ef2b43961b47209f0b754608ebbc0b: stopping tablet replica
I20260709 13:06:31.772231  7481 master.cc:584] Master@127.7.78.126:34015 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (2947 ms)
[----------] 6 tests from LeaderRebalancerTest (360232 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 13:06:31.809310  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:40605
I20260709 13:06:31.810611  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:31.816076 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:31.816558 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:31.818370 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:31.819394  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:31.820317  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:31.820502  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:31.820693  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602391820678 us; error 0 us; skew 500 ppm
I20260709 13:06:31.821238  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:31.823855  7481 webserver.cc:533] Webserver started at http://127.7.78.126:45355/ using document root <none> and password file <none>
I20260709 13:06:31.824390  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:31.824589  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:31.824971  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:31.826224  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "3b5c046b9f3344a08ae2144554581019"
format_stamp: "Formatted at 2026-07-09 13:06:31 on dist-test-slave-50cs"
I20260709 13:06:31.831168  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:06:31.835287 19217 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:31.836364  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 13:06:31.836678  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "3b5c046b9f3344a08ae2144554581019"
format_stamp: "Formatted at 2026-07-09 13:06:31 on dist-test-slave-50cs"
I20260709 13:06:31.836971  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:31.850323  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:31.851320  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:31.893769  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:40605
I20260709 13:06:31.893903 19268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:40605 every 8 connection(s)
I20260709 13:06:31.898149 19269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:31.911427 19269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019: Bootstrap starting.
I20260709 13:06:31.916138 19269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:31.920830 19269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019: No bootstrap required, opened a new log
I20260709 13:06:31.923143 19269 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5c046b9f3344a08ae2144554581019" member_type: VOTER }
I20260709 13:06:31.923550 19269 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:31.923857 19269 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5c046b9f3344a08ae2144554581019, State: Initialized, Role: FOLLOWER
I20260709 13:06:31.924532 19269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5c046b9f3344a08ae2144554581019" member_type: VOTER }
I20260709 13:06:31.925030 19269 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:31.925271 19269 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:31.925580 19269 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:31.930548 19269 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5c046b9f3344a08ae2144554581019" member_type: VOTER }
I20260709 13:06:31.931113 19269 leader_election.cc:304] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b5c046b9f3344a08ae2144554581019; no voters: 
I20260709 13:06:31.932289 19269 leader_election.cc:290] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:31.932684 19272 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:31.934203 19272 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5c046b9f3344a08ae2144554581019, State: Running, Role: LEADER
I20260709 13:06:31.934882 19272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5c046b9f3344a08ae2144554581019" member_type: VOTER }
I20260709 13:06:31.935170 19269 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:31.937842 19274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b5c046b9f3344a08ae2144554581019. Latest consensus state: current_term: 1 leader_uuid: "3b5c046b9f3344a08ae2144554581019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5c046b9f3344a08ae2144554581019" member_type: VOTER } }
I20260709 13:06:31.937808 19273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b5c046b9f3344a08ae2144554581019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5c046b9f3344a08ae2144554581019" member_type: VOTER } }
I20260709 13:06:31.938623 19274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:31.938644 19273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:31.942597 19277 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:31.947557 19277 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:31.952370  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:31.957031 19277 catalog_manager.cc:1383] Generated new cluster ID: c5262306e98843ffb54e221290fbecca
I20260709 13:06:31.957386 19277 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:31.987100 19277 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:31.988560 19277 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:32.020651 19277 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019: Generated new TSK 0
I20260709 13:06:32.021351 19277 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:32.083995  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:32.090857 19290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:32.092231 19291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:32.093703 19293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:32.094571  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:32.100772  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:32.101064  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:32.101251  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602392101234 us; error 0 us; skew 500 ppm
I20260709 13:06:32.101859  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:32.104455  7481 webserver.cc:533] Webserver started at http://127.7.78.65:43113/ using document root <none> and password file <none>
I20260709 13:06:32.105026  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:32.105229  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:32.105552  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:32.106689  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "10584038f61a4dd4a618aa4d75689a05"
format_stamp: "Formatted at 2026-07-09 13:06:32 on dist-test-slave-50cs"
I20260709 13:06:32.111377  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 13:06:32.115170 19298 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.116165  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 13:06:32.116542  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "10584038f61a4dd4a618aa4d75689a05"
format_stamp: "Formatted at 2026-07-09 13:06:32 on dist-test-slave-50cs"
I20260709 13:06:32.116964  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:32.141993  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:32.143235  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:32.144889  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:32.147495  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:32.147782  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.148051  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:32.148244  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.191274  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:37185
I20260709 13:06:32.191398 19360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:37185 every 8 connection(s)
I20260709 13:06:32.196971  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:32.204632 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:32.205011 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:32.209800 19369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:32.210844 19361 heartbeater.cc:344] Connected to a master server at 127.7.78.126:40605
I20260709 13:06:32.210925  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:32.211490 19361 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:32.212116  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:32.212369  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:32.212368 19361 heartbeater.cc:507] Master 127.7.78.126:40605 requested a full tablet report, sending...
I20260709 13:06:32.212797  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602392212775 us; error 0 us; skew 500 ppm
I20260709 13:06:32.213555  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:32.215188 19234 ts_manager.cc:194] Registered new tserver with Master: 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185)
I20260709 13:06:32.216612  7481 webserver.cc:533] Webserver started at http://127.7.78.66:34715/ using document root <none> and password file <none>
I20260709 13:06:32.217211  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:32.217478  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:32.217554 19234 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42382
I20260709 13:06:32.217763  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:32.219341  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
format_stamp: "Formatted at 2026-07-09 13:06:32 on dist-test-slave-50cs"
I20260709 13:06:32.224184  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20260709 13:06:32.228035 19373 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.229059  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:06:32.229352  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
format_stamp: "Formatted at 2026-07-09 13:06:32 on dist-test-slave-50cs"
I20260709 13:06:32.229681  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:32.242184  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:32.243269  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:32.245319  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:32.247771  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:32.247963  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.248307  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:32.248450  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.295881  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:41545
I20260709 13:06:32.296001 19435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:41545 every 8 connection(s)
I20260709 13:06:32.300829  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:32.309372 19439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:32.311246 19440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:32.315006  7481 server_base.cc:1061] running on GCE node
W20260709 13:06:32.315157 19442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:32.315649 19436 heartbeater.cc:344] Connected to a master server at 127.7.78.126:40605
I20260709 13:06:32.316200 19436 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:32.316572  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:32.316880  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:32.316974 19436 heartbeater.cc:507] Master 127.7.78.126:40605 requested a full tablet report, sending...
I20260709 13:06:32.317029  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602392317016 us; error 0 us; skew 500 ppm
I20260709 13:06:32.318063  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:32.319818 19234 ts_manager.cc:194] Registered new tserver with Master: c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:32.321497  7481 webserver.cc:533] Webserver started at http://127.7.78.67:46723/ using document root <none> and password file <none>
I20260709 13:06:32.321997 19234 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42392
I20260709 13:06:32.322294  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:32.322649  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:32.323007  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:32.324524  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "2a776fc68e534b9ea0f748905e1de9e8"
format_stamp: "Formatted at 2026-07-09 13:06:32 on dist-test-slave-50cs"
I20260709 13:06:32.329941  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:06:32.334290 19447 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.335493  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 13:06:32.335952  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "2a776fc68e534b9ea0f748905e1de9e8"
format_stamp: "Formatted at 2026-07-09 13:06:32 on dist-test-slave-50cs"
I20260709 13:06:32.336304  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:32.366595  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:32.367820  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:32.369582  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:32.372088  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:32.372360  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.372671  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:32.372902  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:32.415709  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:35297
I20260709 13:06:32.415886 19509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:35297 every 8 connection(s)
I20260709 13:06:32.433735 19510 heartbeater.cc:344] Connected to a master server at 127.7.78.126:40605
I20260709 13:06:32.434314 19510 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:32.435231 19510 heartbeater.cc:507] Master 127.7.78.126:40605 requested a full tablet report, sending...
I20260709 13:06:32.438297 19234 ts_manager.cc:194] Registered new tserver with Master: 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:32.439100  7481 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018759067s
I20260709 13:06:32.440205 19234 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42408
I20260709 13:06:32.470467 19234 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42416:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:32.473651 19234 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:32.563704 19475 tablet_service.cc:1511] Processing CreateTablet for tablet 4e07978d5df643538c4d45a6a7ae4f47 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:32.565738 19475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e07978d5df643538c4d45a6a7ae4f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.572494 19474 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd0066d193143238a07dac629a958b7 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:32.574136 19474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd0066d193143238a07dac629a958b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.578528 19473 tablet_service.cc:1511] Processing CreateTablet for tablet fc4f89a87a274604883f9b4f91d9872b (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:32.580168 19473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc4f89a87a274604883f9b4f91d9872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.582562 19472 tablet_service.cc:1511] Processing CreateTablet for tablet 468cc51b6ba9441d8a566322b6e6bcbc (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:32.583289 19468 tablet_service.cc:1511] Processing CreateTablet for tablet 26edbcff77f1411481183330d339edd8 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:32.584197 19472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 468cc51b6ba9441d8a566322b6e6bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.585673 19469 tablet_service.cc:1511] Processing CreateTablet for tablet eb10d732a4834e1cbd9d4eb315bcb0dd (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:32.587248 19469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb10d732a4834e1cbd9d4eb315bcb0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.586983 19401 tablet_service.cc:1511] Processing CreateTablet for tablet 4e07978d5df643538c4d45a6a7ae4f47 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:32.588460 19401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e07978d5df643538c4d45a6a7ae4f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.589833 19468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26edbcff77f1411481183330d339edd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.590590 19470 tablet_service.cc:1511] Processing CreateTablet for tablet e05e4b201969463ea0a99d83676e1b23 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:32.592051 19470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e05e4b201969463ea0a99d83676e1b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.595160 19471 tablet_service.cc:1511] Processing CreateTablet for tablet a971384ea43f45c0a2753de28bbad566 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:32.596825 19471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a971384ea43f45c0a2753de28bbad566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.598538 19467 tablet_service.cc:1511] Processing CreateTablet for tablet b8afd9e635b348b0938920cd55a0615a (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:32.600485 19467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8afd9e635b348b0938920cd55a0615a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.600540 19400 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd0066d193143238a07dac629a958b7 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:32.603150 19400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd0066d193143238a07dac629a958b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.612231 19399 tablet_service.cc:1511] Processing CreateTablet for tablet fc4f89a87a274604883f9b4f91d9872b (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:32.615286 19399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc4f89a87a274604883f9b4f91d9872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.628149 19398 tablet_service.cc:1511] Processing CreateTablet for tablet 468cc51b6ba9441d8a566322b6e6bcbc (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:32.630801 19398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 468cc51b6ba9441d8a566322b6e6bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.646152 19321 tablet_service.cc:1511] Processing CreateTablet for tablet e05e4b201969463ea0a99d83676e1b23 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:32.648357 19321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e05e4b201969463ea0a99d83676e1b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.651093 19326 tablet_service.cc:1511] Processing CreateTablet for tablet 4e07978d5df643538c4d45a6a7ae4f47 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:32.655057 19325 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd0066d193143238a07dac629a958b7 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:32.658344 19324 tablet_service.cc:1511] Processing CreateTablet for tablet fc4f89a87a274604883f9b4f91d9872b (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:32.661641 19323 tablet_service.cc:1511] Processing CreateTablet for tablet 468cc51b6ba9441d8a566322b6e6bcbc (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:32.665019 19322 tablet_service.cc:1511] Processing CreateTablet for tablet a971384ea43f45c0a2753de28bbad566 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:32.669387 19320 tablet_service.cc:1511] Processing CreateTablet for tablet eb10d732a4834e1cbd9d4eb315bcb0dd (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:32.673431 19319 tablet_service.cc:1511] Processing CreateTablet for tablet 26edbcff77f1411481183330d339edd8 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:32.653338 19326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e07978d5df643538c4d45a6a7ae4f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.676095 19322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a971384ea43f45c0a2753de28bbad566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.682830 19397 tablet_service.cc:1511] Processing CreateTablet for tablet a971384ea43f45c0a2753de28bbad566 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:32.684949 19397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a971384ea43f45c0a2753de28bbad566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.687665 19396 tablet_service.cc:1511] Processing CreateTablet for tablet e05e4b201969463ea0a99d83676e1b23 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:32.691625 19395 tablet_service.cc:1511] Processing CreateTablet for tablet eb10d732a4834e1cbd9d4eb315bcb0dd (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:32.694783 19319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26edbcff77f1411481183330d339edd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.698060 19324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc4f89a87a274604883f9b4f91d9872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.702133 19325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd0066d193143238a07dac629a958b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.709959 19323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 468cc51b6ba9441d8a566322b6e6bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.711305 19396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e05e4b201969463ea0a99d83676e1b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.709118 19525 tablet_bootstrap.cc:492] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:32.713815 19320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb10d732a4834e1cbd9d4eb315bcb0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.725948 19395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb10d732a4834e1cbd9d4eb315bcb0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.715888 19318 tablet_service.cc:1511] Processing CreateTablet for tablet b8afd9e635b348b0938920cd55a0615a (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:32.718191 19394 tablet_service.cc:1511] Processing CreateTablet for tablet 26edbcff77f1411481183330d339edd8 (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:32.720167 19393 tablet_service.cc:1511] Processing CreateTablet for tablet b8afd9e635b348b0938920cd55a0615a (DEFAULT_TABLE table=test-workload [id=db564bd7ff2b48298d526129956048bd]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:32.728150 19318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8afd9e635b348b0938920cd55a0615a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.733815 19393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8afd9e635b348b0938920cd55a0615a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.737234 19394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26edbcff77f1411481183330d339edd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:32.779143 19525 tablet_bootstrap.cc:654] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.785596 19526 tablet_bootstrap.cc:492] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:32.798164 19526 tablet_bootstrap.cc:654] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.819191 19527 tablet_bootstrap.cc:492] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:32.826089 19527 tablet_bootstrap.cc:654] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.830369 19526 tablet_bootstrap.cc:492] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:32.830892 19526 ts_tablet_manager.cc:1403] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.046s	user 0.013s	sys 0.014s
I20260709 13:06:32.834007 19526 raft_consensus.cc:359] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.834839 19526 raft_consensus.cc:385] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:32.835137 19526 raft_consensus.cc:740] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:32.836045 19526 consensus_queue.cc:260] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.838279 19361 heartbeater.cc:499] Master 127.7.78.126:40605 was elected leader, sending a full tablet report...
I20260709 13:06:32.838893 19526 ts_tablet_manager.cc:1434] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:32.839990 19526 tablet_bootstrap.cc:492] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:32.841775 19527 tablet_bootstrap.cc:492] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:32.842634 19527 ts_tablet_manager.cc:1403] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.000s
I20260709 13:06:32.846513 19526 tablet_bootstrap.cc:654] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.847522 19527 raft_consensus.cc:359] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:32.848727 19527 raft_consensus.cc:385] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:32.849110 19527 raft_consensus.cc:740] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:32.850489 19527 consensus_queue.cc:260] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:32.857352 19533 raft_consensus.cc:493] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:32.857483 19436 heartbeater.cc:499] Master 127.7.78.126:40605 was elected leader, sending a full tablet report...
I20260709 13:06:32.863473 19525 tablet_bootstrap.cc:492] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:32.864117 19525 ts_tablet_manager.cc:1403] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.155s	user 0.030s	sys 0.019s
I20260709 13:06:32.867863 19527 ts_tablet_manager.cc:1434] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.008s
I20260709 13:06:32.860929 19533 raft_consensus.cc:515] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:32.869127 19527 tablet_bootstrap.cc:492] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:32.875841 19527 tablet_bootstrap.cc:654] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.879374 19526 tablet_bootstrap.cc:492] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:32.879602 19533 leader_election.cc:290] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297), 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185)
I20260709 13:06:32.867450 19525 raft_consensus.cc:359] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:32.880188 19526 ts_tablet_manager.cc:1403] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.040s	user 0.024s	sys 0.014s
I20260709 13:06:32.880401 19525 raft_consensus.cc:385] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:32.880741 19525 raft_consensus.cc:740] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:32.881724 19525 consensus_queue.cc:260] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:32.884074 19526 raft_consensus.cc:359] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.885355 19526 raft_consensus.cc:385] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:32.885888 19526 raft_consensus.cc:740] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:32.887341 19526 consensus_queue.cc:260] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.896255 19526 ts_tablet_manager.cc:1434] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:06:32.898087 19526 tablet_bootstrap.cc:492] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:32.907285 19526 tablet_bootstrap.cc:654] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.939700 19510 heartbeater.cc:499] Master 127.7.78.126:40605 was elected leader, sending a full tablet report...
I20260709 13:06:32.940270 19525 ts_tablet_manager.cc:1434] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.076s	user 0.033s	sys 0.028s
I20260709 13:06:32.941583 19525 tablet_bootstrap.cc:492] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:32.941758 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:32.942545 19485 raft_consensus.cc:2468] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
W20260709 13:06:32.947355 19282 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:32.947715 19525 tablet_bootstrap.cc:654] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.948395 19336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:32.949715 19376 leader_election.cc:304] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
W20260709 13:06:32.950390 19375 leader_election.cc:343] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:32.950520 19533 raft_consensus.cc:2804] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:32.951013 19533 raft_consensus.cc:493] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:32.951382 19533 raft_consensus.cc:3060] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:32.951629 19527 tablet_bootstrap.cc:492] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:32.952227 19527 ts_tablet_manager.cc:1403] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.083s	user 0.019s	sys 0.020s
I20260709 13:06:32.953553 19526 tablet_bootstrap.cc:492] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:32.954164 19526 ts_tablet_manager.cc:1403] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.056s	user 0.016s	sys 0.012s
I20260709 13:06:32.955441 19527 raft_consensus.cc:359] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.956360 19527 raft_consensus.cc:385] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:32.956698 19527 raft_consensus.cc:740] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:32.957429 19526 raft_consensus.cc:359] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.957762 19527 consensus_queue.cc:260] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.958365 19526 raft_consensus.cc:385] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:32.958833 19526 raft_consensus.cc:740] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:32.959714 19526 consensus_queue.cc:260] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.961192 19533 raft_consensus.cc:515] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:32.962323 19526 ts_tablet_manager.cc:1434] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:32.963698 19533 leader_election.cc:290] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297), 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185)
I20260709 13:06:32.964612 19526 tablet_bootstrap.cc:492] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:32.964828 19336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:32.965019 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:32.965579 19485 raft_consensus.cc:3060] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:32.966465 19375 leader_election.cc:343] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:32.968405 19527 ts_tablet_manager.cc:1434] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 13:06:32.969574 19527 tablet_bootstrap.cc:492] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:32.974179 19531 raft_consensus.cc:493] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:32.974289 19485 raft_consensus.cc:2468] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:32.975526 19527 tablet_bootstrap.cc:654] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.974880 19531 raft_consensus.cc:515] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.978780 19525 tablet_bootstrap.cc:492] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:32.979453 19525 ts_tablet_manager.cc:1403] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.038s	user 0.016s	sys 0.007s
I20260709 13:06:32.978900 19376 leader_election.cc:304] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:32.981945 19533 raft_consensus.cc:2804] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:32.982453 19533 raft_consensus.cc:697] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:32.982614 19525 raft_consensus.cc:359] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.983453 19525 raft_consensus.cc:385] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:32.983855 19525 raft_consensus.cc:740] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:32.984742 19525 consensus_queue.cc:260] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:32.987354 19525 ts_tablet_manager.cc:1434] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:32.987541 19531 leader_election.cc:290] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:32.988564 19525 tablet_bootstrap.cc:492] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:32.983323 19533 consensus_queue.cc:237] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:32.997174 19525 tablet_bootstrap.cc:654] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:32.974766 19526 tablet_bootstrap.cc:654] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.008069 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.008391 19233 catalog_manager.cc:5697] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:33.010100 19300 leader_election.cc:343] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.014305 19525 tablet_bootstrap.cc:492] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:33.014305 19526 tablet_bootstrap.cc:492] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:33.015008 19525 ts_tablet_manager.cc:1403] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.027s	user 0.021s	sys 0.004s
I20260709 13:06:33.015108 19526 ts_tablet_manager.cc:1403] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.051s	user 0.015s	sys 0.000s
I20260709 13:06:33.018157 19525 raft_consensus.cc:359] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.018455 19526 raft_consensus.cc:359] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:33.018980 19525 raft_consensus.cc:385] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.019369 19525 raft_consensus.cc:740] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.019390 19526 raft_consensus.cc:385] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.019937 19526 raft_consensus.cc:740] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.020437 19525 consensus_queue.cc:260] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.020895 19526 consensus_queue.cc:260] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:33.023480 19526 ts_tablet_manager.cc:1434] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 13:06:33.023624 19527 tablet_bootstrap.cc:492] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:33.024154 19527 ts_tablet_manager.cc:1403] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.055s	user 0.010s	sys 0.007s
I20260709 13:06:33.024544 19526 tablet_bootstrap.cc:492] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:33.024564 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.027165 19527 raft_consensus.cc:359] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.027935 19527 raft_consensus.cc:385] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.028229 19527 raft_consensus.cc:740] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
W20260709 13:06:33.028330 19301 leader_election.cc:343] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.028854 19301 leader_election.cc:304] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05; no voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.029057 19527 consensus_queue.cc:260] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.030083 19525 ts_tablet_manager.cc:1434] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.004s
I20260709 13:06:33.030179 19531 raft_consensus.cc:2749] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.031526 19527 ts_tablet_manager.cc:1434] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 13:06:33.032838 19527 tablet_bootstrap.cc:492] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:33.034070 19525 tablet_bootstrap.cc:492] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:33.041558 19527 tablet_bootstrap.cc:654] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.041719 19526 tablet_bootstrap.cc:654] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.042718 19525 tablet_bootstrap.cc:654] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.054324 19527 tablet_bootstrap.cc:492] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:33.054795 19525 tablet_bootstrap.cc:492] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:33.054844 19527 ts_tablet_manager.cc:1403] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.022s	user 0.007s	sys 0.011s
I20260709 13:06:33.055821 19525 ts_tablet_manager.cc:1403] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.004s
I20260709 13:06:33.058364 19527 raft_consensus.cc:359] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.059059 19527 raft_consensus.cc:385] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.059291 19526 tablet_bootstrap.cc:492] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:33.059319 19527 raft_consensus.cc:740] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.059938 19526 ts_tablet_manager.cc:1403] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.036s	user 0.019s	sys 0.004s
I20260709 13:06:33.059340 19525 raft_consensus.cc:359] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.060835 19525 raft_consensus.cc:385] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.060820 19527 consensus_queue.cc:260] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.061362 19525 raft_consensus.cc:740] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.062426 19525 consensus_queue.cc:260] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.063386 19527 ts_tablet_manager.cc:1434] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:33.063464 19526 raft_consensus.cc:359] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.064630 19527 tablet_bootstrap.cc:492] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:33.066576 19525 ts_tablet_manager.cc:1434] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:06:33.067854 19525 tablet_bootstrap.cc:492] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:33.064589 19526 raft_consensus.cc:385] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.069582 19526 raft_consensus.cc:740] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.070412 19526 consensus_queue.cc:260] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.073235 19526 ts_tablet_manager.cc:1434] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 13:06:33.074393 19526 tablet_bootstrap.cc:492] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:33.076108 19525 tablet_bootstrap.cc:654] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.078099 19533 raft_consensus.cc:493] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.078715 19533 raft_consensus.cc:515] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.082675 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.084517 19527 tablet_bootstrap.cc:654] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.084995 19526 tablet_bootstrap.cc:654] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.085366 19336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.086563 19336 raft_consensus.cc:2468] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:33.087162 19525 tablet_bootstrap.cc:492] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:33.087886 19525 ts_tablet_manager.cc:1403] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.012s
I20260709 13:06:33.092118 19525 raft_consensus.cc:359] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.093561 19525 raft_consensus.cc:385] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:06:33.093999 19376 leader_election.cc:343] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.094138 19525 raft_consensus.cc:740] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.096027 19375 leader_election.cc:304] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:33.096192 19525 consensus_queue.cc:260] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.097647 19534 raft_consensus.cc:2804] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.098313 19534 raft_consensus.cc:493] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.098798 19534 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.100713 19525 ts_tablet_manager.cc:1434] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.003s
I20260709 13:06:33.102643 19525 tablet_bootstrap.cc:492] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:33.110812 19533 leader_election.cc:290] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.112751 19525 tablet_bootstrap.cc:654] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.127218 19525 tablet_bootstrap.cc:492] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:33.128129 19525 ts_tablet_manager.cc:1403] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.012s
I20260709 13:06:33.129742 19538 raft_consensus.cc:493] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.130554 19538 raft_consensus.cc:515] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.134284 19538 leader_election.cc:290] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.136727 19534 raft_consensus.cc:515] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.144826 19531 raft_consensus.cc:493] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.145457 19531 raft_consensus.cc:515] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.146924 19534 leader_election.cc:290] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.142202 19525 raft_consensus.cc:359] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.149227 19525 raft_consensus.cc:385] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.149713 19525 raft_consensus.cc:740] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.150094 19336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.150781 19336 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.151048 19526 tablet_bootstrap.cc:492] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:33.151872 19526 ts_tablet_manager.cc:1403] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.078s	user 0.031s	sys 0.040s
I20260709 13:06:33.150907 19525 consensus_queue.cc:260] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.154368 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.154699 19525 ts_tablet_manager.cc:1434] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260709 13:06:33.155193 19526 raft_consensus.cc:359] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.155599 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.156061 19526 raft_consensus.cc:385] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.156323 19525 tablet_bootstrap.cc:492] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:33.156391 19526 raft_consensus.cc:740] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
W20260709 13:06:33.157647 19300 leader_election.cc:343] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.157694 19526 consensus_queue.cc:260] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.155192 19485 raft_consensus.cc:2468] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 0.
I20260709 13:06:33.160414 19301 leader_election.cc:304] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.160692 19538 raft_consensus.cc:493] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.161360 19534 raft_consensus.cc:493] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.156662 19531 leader_election.cc:290] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.161423 19538 raft_consensus.cc:515] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.162115 19534 raft_consensus.cc:515] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.164186 19538 leader_election.cc:290] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.165503 19534 leader_election.cc:290] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.166107 19525 tablet_bootstrap.cc:654] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.165663 19557 raft_consensus.cc:2804] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.166533 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
W20260709 13:06:33.168648 19375 leader_election.cc:343] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.169359 19557 raft_consensus.cc:493] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.170455 19557 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.182129 19526 ts_tablet_manager.cc:1434] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.030s	user 0.024s	sys 0.002s
I20260709 13:06:33.183349 19526 tablet_bootstrap.cc:492] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:33.187057 19525 tablet_bootstrap.cc:492] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:33.187906 19525 ts_tablet_manager.cc:1403] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.032s	user 0.024s	sys 0.004s
I20260709 13:06:33.190325 19526 tablet_bootstrap.cc:654] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.191857 19531 raft_consensus.cc:493] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.192556 19556 raft_consensus.cc:493] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.194012 19556 raft_consensus.cc:515] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:33.195626 19525 raft_consensus.cc:359] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.196446 19525 raft_consensus.cc:385] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.196736 19525 raft_consensus.cc:740] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.197479 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.195677 19527 tablet_bootstrap.cc:492] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:33.198099 19557 raft_consensus.cc:515] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.198516 19527 ts_tablet_manager.cc:1403] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.134s	user 0.037s	sys 0.028s
I20260709 13:06:33.202013 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.202538 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.202628 19485 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.192811 19531 raft_consensus.cc:515] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
W20260709 13:06:33.203909 19300 leader_election.cc:343] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.192502 19336 raft_consensus.cc:2468] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:33.205458 19375 leader_election.cc:304] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:33.205801 19531 leader_election.cc:290] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.206780 19556 leader_election.cc:290] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.207844 19484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.197544 19525 consensus_queue.cc:260] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.208567 19484 raft_consensus.cc:2468] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 0.
I20260709 13:06:33.206820 19534 raft_consensus.cc:2804] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.209847 19534 raft_consensus.cc:697] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:33.210682 19525 ts_tablet_manager.cc:1434] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.006s
I20260709 13:06:33.212227 19525 tablet_bootstrap.cc:492] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:33.214514 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.210722 19534 consensus_queue.cc:237] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.212549 19301 leader_election.cc:304] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.218022 19484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.219241 19484 raft_consensus.cc:2393] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10584038f61a4dd4a618aa4d75689a05 in current term 1: Already voted for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in this term.
I20260709 13:06:33.219570 19485 raft_consensus.cc:2468] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 1.
I20260709 13:06:33.221136 19301 leader_election.cc:304] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05; no voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.203006 19527 raft_consensus.cc:359] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.222034 19527 raft_consensus.cc:385] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.222518 19527 raft_consensus.cc:740] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.222797 19301 leader_election.cc:304] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.223738 19527 consensus_queue.cc:260] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.224640 19531 raft_consensus.cc:2804] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.225227 19531 raft_consensus.cc:493] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.225723 19531 raft_consensus.cc:3060] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.227676 19525 tablet_bootstrap.cc:654] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.229980 19411 raft_consensus.cc:2468] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 0.
I20260709 13:06:33.236320 19556 raft_consensus.cc:3060] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.237515 19527 ts_tablet_manager.cc:1434] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.038s	user 0.009s	sys 0.002s
I20260709 13:06:33.237740 19526 tablet_bootstrap.cc:492] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:33.238282 19526 ts_tablet_manager.cc:1403] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.055s	user 0.024s	sys 0.029s
I20260709 13:06:33.241645 19526 raft_consensus.cc:359] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.242223 19527 tablet_bootstrap.cc:492] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:33.242365 19526 raft_consensus.cc:385] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.242775 19526 raft_consensus.cc:740] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.243815 19526 consensus_queue.cc:260] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.256323 19531 raft_consensus.cc:515] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.258404 19563 raft_consensus.cc:2804] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.259433 19526 ts_tablet_manager.cc:1434] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.015s
I20260709 13:06:33.260663 19526 tablet_bootstrap.cc:492] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:33.203151 19557 leader_election.cc:290] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.267850 19526 tablet_bootstrap.cc:654] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.269153 19527 tablet_bootstrap.cc:654] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.277473 19563 raft_consensus.cc:697] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.276466 19234 catalog_manager.cc:5697] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.278538 19563 consensus_queue.cc:237] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
W20260709 13:06:33.282596 19301 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.67:35297, user_credentials={real_user=slave}} blocked reactor thread for 59952.4us
I20260709 13:06:33.283974 19526 tablet_bootstrap.cc:492] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:33.285128 19526 ts_tablet_manager.cc:1403] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.004s
I20260709 13:06:33.287184 19531 leader_election.cc:290] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.288086 19526 raft_consensus.cc:359] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.288865 19526 raft_consensus.cc:385] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.289121 19526 raft_consensus.cc:740] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.289611 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.290385 19411 raft_consensus.cc:3060] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.290171 19526 consensus_queue.cc:260] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.292472 19526 ts_tablet_manager.cc:1434] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:06:33.293543 19526 tablet_bootstrap.cc:492] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:33.293546 19534 raft_consensus.cc:493] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.294142 19534 raft_consensus.cc:515] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.296494 19534 leader_election.cc:290] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.298883 19336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.300911 19526 tablet_bootstrap.cc:654] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:33.301303 19375 leader_election.cc:343] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.303465 19411 raft_consensus.cc:2468] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 1.
I20260709 13:06:33.304860 19300 leader_election.cc:304] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:33.305583 19531 raft_consensus.cc:2804] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.306066 19531 raft_consensus.cc:697] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.306866 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.307569 19485 raft_consensus.cc:3060] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.306893 19531 consensus_queue.cc:237] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.313577 19564 raft_consensus.cc:493] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.314260 19564 raft_consensus.cc:515] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.314806 19538 raft_consensus.cc:493] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.316272 19538 raft_consensus.cc:515] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.318601 19538 leader_election.cc:290] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.318593 19527 tablet_bootstrap.cc:492] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:33.320967 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.321548 19527 ts_tablet_manager.cc:1403] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.080s	user 0.019s	sys 0.002s
I20260709 13:06:33.323474 19525 tablet_bootstrap.cc:492] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:33.324090 19556 raft_consensus.cc:2749] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:06:33.324218 19300 leader_election.cc:343] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.324774 19484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.324159 19525 ts_tablet_manager.cc:1403] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.112s	user 0.013s	sys 0.023s
I20260709 13:06:33.325501 19484 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.325426 19483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.327212 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.327898 19482 raft_consensus.cc:2468] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:33.328545 19525 raft_consensus.cc:359] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
W20260709 13:06:33.329864 19376 leader_election.cc:343] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.330538 19376 leader_election.cc:304] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:33.332333 19336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.333192 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.329381 19525 raft_consensus.cc:385] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.335089 19376 leader_election.cc:304] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:33.334064 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.335857 19410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.336135 19533 raft_consensus.cc:2804] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.335039 19525 raft_consensus.cc:740] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.336673 19533 raft_consensus.cc:493] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.338114 19533 raft_consensus.cc:3060] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.338019 19527 raft_consensus.cc:359] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.338667 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.338881 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.334744 19534 raft_consensus.cc:2749] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.338317 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.334080 19335 raft_consensus.cc:2393] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a776fc68e534b9ea0f748905e1de9e8 in current term 1: Already voted for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in this term.
I20260709 13:06:33.339192 19527 raft_consensus.cc:385] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.341290 19527 raft_consensus.cc:740] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.342406 19527 consensus_queue.cc:260] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
W20260709 13:06:33.346400 19449 leader_election.cc:343] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:33.349534 19449 leader_election.cc:343] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.350046 19449 leader_election.cc:304] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8; no voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.349885 19525 consensus_queue.cc:260] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.350898 19538 raft_consensus.cc:2749] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.350929 19482 raft_consensus.cc:2468] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 0.
I20260709 13:06:33.354101 19301 leader_election.cc:304] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.356585 19485 raft_consensus.cc:2468] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 1.
I20260709 13:06:33.356805 19563 raft_consensus.cc:2804] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.330004 19564 leader_election.cc:290] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.365494 19484 raft_consensus.cc:2468] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:33.359930 19563 raft_consensus.cc:493] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.367022 19563 raft_consensus.cc:3060] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.367421 19525 ts_tablet_manager.cc:1434] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.042s	user 0.010s	sys 0.000s
I20260709 13:06:33.368753 19525 tablet_bootstrap.cc:492] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:33.368753 19232 catalog_manager.cc:5697] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.360561 19449 leader_election.cc:304] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8; no voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.374181 19525 tablet_bootstrap.cc:654] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.376080 19538 raft_consensus.cc:2749] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.375696 19527 ts_tablet_manager.cc:1434] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.054s	user 0.008s	sys 0.000s
I20260709 13:06:33.377753 19527 tablet_bootstrap.cc:492] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:33.364257 19533 raft_consensus.cc:515] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.381484 19408 raft_consensus.cc:2393] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a776fc68e534b9ea0f748905e1de9e8 in current term 1: Already voted for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in this term.
I20260709 13:06:33.383677 19563 raft_consensus.cc:515] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.387763 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.389214 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.389454 19563 leader_election.cc:290] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.390029 19485 raft_consensus.cc:3060] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.390825 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
W20260709 13:06:33.390995 19375 leader_election.cc:343] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:33.392505 19300 leader_election.cc:343] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.392753 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.393481 19482 raft_consensus.cc:3060] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.388743 19449 leader_election.cc:304] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8; no voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.396509 19526 tablet_bootstrap.cc:492] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:33.397089 19526 ts_tablet_manager.cc:1403] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.104s	user 0.020s	sys 0.011s
I20260709 13:06:33.396270 19534 consensus_queue.cc:1048] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.399322 19485 raft_consensus.cc:2468] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:33.400044 19538 raft_consensus.cc:2749] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.400146 19526 raft_consensus.cc:359] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.400980 19526 raft_consensus.cc:385] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.401284 19526 raft_consensus.cc:740] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.402024 19526 consensus_queue.cc:260] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.404269 19526 ts_tablet_manager.cc:1434] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:06:33.406255 19376 leader_election.cc:304] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:33.407015 19527 tablet_bootstrap.cc:654] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.407320 19534 raft_consensus.cc:2804] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.408887 19525 tablet_bootstrap.cc:492] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:33.409531 19525 ts_tablet_manager.cc:1403] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.041s	user 0.012s	sys 0.007s
I20260709 13:06:33.413599 19525 raft_consensus.cc:359] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.416045 19525 raft_consensus.cc:385] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.416364 19525 raft_consensus.cc:740] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.417198 19525 consensus_queue.cc:260] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.420105 19525 ts_tablet_manager.cc:1434] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20260709 13:06:33.385918 19533 leader_election.cc:290] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.417042 19233 catalog_manager.cc:5697] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.426399 19482 raft_consensus.cc:2468] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 1.
I20260709 13:06:33.428032 19301 leader_election.cc:304] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.430040 19563 raft_consensus.cc:2804] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.430722 19563 raft_consensus.cc:697] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.431697 19534 raft_consensus.cc:697] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:33.431732 19563 consensus_queue.cc:237] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.432574 19534 consensus_queue.cc:237] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.435173 19527 tablet_bootstrap.cc:492] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:33.441748 19527 ts_tablet_manager.cc:1403] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.064s	user 0.023s	sys 0.000s
I20260709 13:06:33.446935 19527 raft_consensus.cc:359] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.448467 19527 raft_consensus.cc:385] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.449045 19527 raft_consensus.cc:740] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.451454 19527 consensus_queue.cc:260] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.454385 19527 ts_tablet_manager.cc:1434] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 13:06:33.455978 19571 consensus_queue.cc:1048] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:33.459026 19527 tablet_bootstrap.cc:492] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:33.460770 19571 raft_consensus.cc:493] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.461386 19571 raft_consensus.cc:515] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.464715 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.468901 19527 tablet_bootstrap.cc:654] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.470414 19538 raft_consensus.cc:493] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.471131 19538 raft_consensus.cc:515] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.475492 19538 leader_election.cc:290] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.469662 19234 catalog_manager.cc:5697] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.485837 19571 leader_election.cc:290] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.488384 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.490455 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.491498 19334 raft_consensus.cc:2468] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:33.496204 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.497527 19482 raft_consensus.cc:2393] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in current term 1: Already voted for candidate 10584038f61a4dd4a618aa4d75689a05 in this term.
I20260709 13:06:33.497354 19449 leader_election.cc:304] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.500432 19376 leader_election.cc:304] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:33.502162 19534 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.503017 19538 raft_consensus.cc:2804] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.503548 19538 raft_consensus.cc:493] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.503953 19538 raft_consensus.cc:3060] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:33.512254 19449 leader_election.cc:343] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:33.515975 19534 raft_consensus.cc:2749] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.514699 19232 catalog_manager.cc:5697] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.520612 19538 raft_consensus.cc:515] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.522776 19563 raft_consensus.cc:493] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.523502 19527 tablet_bootstrap.cc:492] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:33.523334 19563 raft_consensus.cc:515] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.524051 19527 ts_tablet_manager.cc:1403] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.069s	user 0.017s	sys 0.016s
I20260709 13:06:33.525617 19563 leader_election.cc:290] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.525544 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.526487 19335 raft_consensus.cc:3060] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.526724 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.527415 19482 raft_consensus.cc:2393] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10584038f61a4dd4a618aa4d75689a05 in current term 1: Already voted for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in this term.
I20260709 13:06:33.527236 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e05e4b201969463ea0a99d83676e1b23" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.528798 19300 leader_election.cc:304] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05; no voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.529510 19563 raft_consensus.cc:3060] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.530648 19527 raft_consensus.cc:359] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.531456 19527 raft_consensus.cc:385] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.531828 19527 raft_consensus.cc:740] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.532640 19527 consensus_queue.cc:260] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.534085 19538 leader_election.cc:290] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.534780 19527 ts_tablet_manager.cc:1434] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:06:33.535447 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.535813 19527 tablet_bootstrap.cc:492] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:33.536051 19408 raft_consensus.cc:3060] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.535842 19563 raft_consensus.cc:2749] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.537591 19335 raft_consensus.cc:2468] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:33.540045 19449 leader_election.cc:304] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.541011 19538 raft_consensus.cc:2804] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.542265 19538 raft_consensus.cc:697] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:33.543308 19538 consensus_queue.cc:237] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.545027 19527 tablet_bootstrap.cc:654] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:33.545488 19408 raft_consensus.cc:2468] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:33.552173 19527 tablet_bootstrap.cc:492] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:33.552723 19527 ts_tablet_manager.cc:1403] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.002s
I20260709 13:06:33.553073 19538 raft_consensus.cc:493] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.553855 19538 raft_consensus.cc:515] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.555943 19527 raft_consensus.cc:359] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.556808 19527 raft_consensus.cc:385] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:33.557153 19527 raft_consensus.cc:740] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:33.557333 19234 catalog_manager.cc:5697] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:33.558272 19527 consensus_queue.cc:260] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.560010 19527 ts_tablet_manager.cc:1434] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
I20260709 13:06:33.562233 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.563071 19335 raft_consensus.cc:2468] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:33.564252 19538 leader_election.cc:290] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.565079 19449 leader_election.cc:304] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.566336 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.566821 19538 raft_consensus.cc:2804] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.567303 19408 raft_consensus.cc:2468] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:33.567320 19538 raft_consensus.cc:493] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.570801 19538 raft_consensus.cc:3060] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.576846 19538 raft_consensus.cc:515] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.577821 19583 raft_consensus.cc:493] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.578430 19583 raft_consensus.cc:515] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.579357 19538 leader_election.cc:290] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.580298 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.580530 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.580814 19583 leader_election.cc:290] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.581092 19335 raft_consensus.cc:3060] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.581843 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:33.582261 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.582499 19411 raft_consensus.cc:2468] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:33.582996 19334 raft_consensus.cc:2468] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:33.581271 19408 raft_consensus.cc:3060] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.585182 19449 leader_election.cc:304] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:33.586338 19583 raft_consensus.cc:2804] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:33.586777 19583 raft_consensus.cc:493] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:33.587153 19583 raft_consensus.cc:3060] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.589529 19335 raft_consensus.cc:2468] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:33.590734 19449 leader_election.cc:304] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.591512 19538 raft_consensus.cc:2804] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.592680 19408 raft_consensus.cc:2468] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:33.592721 19538 raft_consensus.cc:697] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:33.594056 19538 consensus_queue.cc:237] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.595693 19583 raft_consensus.cc:515] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.599048 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.599615 19335 raft_consensus.cc:3060] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.599957 19583 leader_election.cc:290] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.600602 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.601197 19408 raft_consensus.cc:3060] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.606925 19335 raft_consensus.cc:2468] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:33.607980 19408 raft_consensus.cc:2468] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:33.608361 19449 leader_election.cc:304] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.609437 19583 raft_consensus.cc:2804] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:33.608458 19234 catalog_manager.cc:5697] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:33.610064 19583 raft_consensus.cc:697] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:33.610868 19583 consensus_queue.cc:237] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.623548 19234 catalog_manager.cc:5697] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:33.696332 19534 consensus_queue.cc:1048] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.697273 19408 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
mode: GRACEFUL
new_leader_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:53720
I20260709 13:06:33.698001 19408 raft_consensus.cc:606] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Received request to transfer leadership to TS 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:33.706130 19408 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0bd0066d193143238a07dac629a958b7"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
mode: GRACEFUL
new_leader_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:53720
I20260709 13:06:33.706863 19408 raft_consensus.cc:606] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Received request to transfer leadership to TS 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:33.710613 19577 consensus_queue.cc:1048] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:33.715375 19577 raft_consensus.cc:993] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28: : Instructing follower 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:33.715945 19577 raft_consensus.cc:1081] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Signalling peer 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:33.718341 19335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0bd0066d193143238a07dac629a958b7"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:57844
I20260709 13:06:33.719249 19335 raft_consensus.cc:493] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:33.719789 19335 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.731600 19335 raft_consensus.cc:515] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.734736 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.735411 19408 raft_consensus.cc:3055] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:33.735778 19408 raft_consensus.cc:740] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:33.735599 19482 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
mode: GRACEFUL
new_leader_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:34882
I20260709 13:06:33.736836 19482 raft_consensus.cc:606] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Received request to transfer leadership to TS 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:33.736922 19335 leader_election.cc:290] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.737469 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.738157 19485 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.736948 19408 consensus_queue.cc:260] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.739840 19408 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.744550 19482 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
mode: GRACEFUL
new_leader_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:34882
I20260709 13:06:33.745277 19482 raft_consensus.cc:606] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Received request to transfer leadership to TS 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:33.746244 19485 raft_consensus.cc:2468] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:33.747670 19301 leader_election.cc:304] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.747844 19408 raft_consensus.cc:2468] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:33.748487 19563 raft_consensus.cc:2804] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:33.748536 19534 raft_consensus.cc:493] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:33.749186 19563 raft_consensus.cc:697] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.750141 19563 consensus_queue.cc:237] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.749213 19534 raft_consensus.cc:515] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.754423 19534 leader_election.cc:290] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.756050 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:33.758200 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:33.759133 19485 raft_consensus.cc:2393] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in current term 1: Already voted for candidate 10584038f61a4dd4a618aa4d75689a05 in this term.
I20260709 13:06:33.759202 19563 consensus_queue.cc:1048] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.760514 19376 leader_election.cc:304] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:33.760116 19233 catalog_manager.cc:5697] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 1 to 2, leader changed from c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66) to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.762141 19534 raft_consensus.cc:3060] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:33.766460 19335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
mode: GRACEFUL
new_leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:57872
I20260709 13:06:33.767292 19335 raft_consensus.cc:606] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Received request to transfer leadership to TS c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.772233 19534 raft_consensus.cc:2749] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:33.778616 19335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
mode: GRACEFUL
new_leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:57872
I20260709 13:06:33.781102 19335 raft_consensus.cc:606] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Received request to transfer leadership to TS c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.783910 19556 consensus_queue.cc:1048] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:33.788473 19563 consensus_queue.cc:1048] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:33.800223 19482 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "26edbcff77f1411481183330d339edd8"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
mode: GRACEFUL
new_leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:34882
I20260709 13:06:33.800892 19482 raft_consensus.cc:606] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Received request to transfer leadership to TS c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:33.807236 19564 raft_consensus.cc:993] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05: : Instructing follower c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:33.807794 19564 raft_consensus.cc:1081] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Signalling peer c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:33.810253 19411 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:53696
I20260709 13:06:33.810859 19411 raft_consensus.cc:493] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:33.811209 19411 raft_consensus.cc:3060] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.811327 19335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
mode: GRACEFUL
new_leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:57872
I20260709 13:06:33.812017 19335 raft_consensus.cc:606] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Received request to transfer leadership to TS 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:33.819242 19556 consensus_queue.cc:1048] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.820166 19411 raft_consensus.cc:515] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.823629 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.824452 19335 raft_consensus.cc:3055] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:33.824802 19335 raft_consensus.cc:740] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.825790 19335 consensus_queue.cc:260] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.826854 19411 leader_election.cc:290] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.826992 19335 raft_consensus.cc:3060] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.827284 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb10d732a4834e1cbd9d4eb315bcb0dd" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.829540 19485 raft_consensus.cc:3060] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.832690 19531 raft_consensus.cc:993] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05: : Instructing follower c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:33.833437 19556 raft_consensus.cc:1081] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Signalling peer c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:33.835481 19408 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:53696
I20260709 13:06:33.836097 19408 raft_consensus.cc:493] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:33.836570 19408 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.838162 19335 raft_consensus.cc:2468] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 2.
I20260709 13:06:33.839649 19375 leader_election.cc:304] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:33.840101 19485 raft_consensus.cc:2468] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 2.
I20260709 13:06:33.840601 19534 raft_consensus.cc:2804] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:33.841189 19534 raft_consensus.cc:697] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:33.842144 19534 consensus_queue.cc:237] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.845144 19408 raft_consensus.cc:515] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.847795 19408 leader_election.cc:290] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.848693 19577 raft_consensus.cc:993] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28: : Instructing follower 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:33.849244 19571 raft_consensus.cc:1081] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Signalling peer 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:33.849714 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.850405 19334 raft_consensus.cc:3055] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:33.850312 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.850910 19334 raft_consensus.cc:740] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.851087 19485 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.851336 19335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:57844
I20260709 13:06:33.852087 19335 raft_consensus.cc:493] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:33.852003 19334 consensus_queue.cc:260] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.852726 19335 raft_consensus.cc:3060] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.853826 19334 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.859604 19335 raft_consensus.cc:515] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:33.861773 19334 raft_consensus.cc:2468] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 2.
I20260709 13:06:33.862910 19335 leader_election.cc:290] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:33.862962 19408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.862236 19485 raft_consensus.cc:2468] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 2.
I20260709 13:06:33.862257 19232 catalog_manager.cc:5697] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 1 to 2, leader changed from 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65) to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.863854 19408 raft_consensus.cc:3055] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:33.864041 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e07978d5df643538c4d45a6a7ae4f47" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:33.864352 19408 raft_consensus.cc:740] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:33.864718 19376 leader_election.cc:304] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:33.865779 19571 raft_consensus.cc:2804] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:33.864790 19482 raft_consensus.cc:3060] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.866302 19571 raft_consensus.cc:697] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:33.867265 19571 consensus_queue.cc:237] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.865305 19408 consensus_queue.cc:260] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:33.868894 19408 raft_consensus.cc:3060] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.875312 19408 raft_consensus.cc:2468] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:33.875774 19482 raft_consensus.cc:2468] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:33.876607 19300 leader_election.cc:304] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:33.877559 19531 raft_consensus.cc:2804] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:33.878027 19531 raft_consensus.cc:697] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.878945 19531 consensus_queue.cc:237] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } }
I20260709 13:06:33.880679 19232 catalog_manager.cc:5697] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 1 to 2, leader changed from 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65) to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:33.887351 19234 catalog_manager.cc:5697] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 1 to 2, leader changed from c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66) to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:33.894976 19571 consensus_queue.cc:1048] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.908085 19571 consensus_queue.cc:1048] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:33.927412 19531 consensus_queue.cc:1048] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:33.941145 19563 consensus_queue.cc:1048] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.947417 19538 consensus_queue.cc:1048] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.951079 19563 raft_consensus.cc:993] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05: : Instructing follower 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:33.951610 19563 raft_consensus.cc:1081] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Signalling peer 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:33.953569 19482 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:34872
I20260709 13:06:33.954324 19482 raft_consensus.cc:493] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:33.954843 19482 raft_consensus.cc:3060] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.962530 19583 raft_consensus.cc:993] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8: : Instructing follower c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:33.963100 19583 raft_consensus.cc:1081] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Signalling peer c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:33.966485 19482 raft_consensus.cc:515] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.968299 19411 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "26edbcff77f1411481183330d339edd8"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:53706
I20260709 13:06:33.969033 19411 raft_consensus.cc:493] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:33.969461 19411 raft_consensus.cc:3060] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.969911 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:33.970607 19334 raft_consensus.cc:3055] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:33.970930 19334 raft_consensus.cc:740] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:33.971062 19482 leader_election.cc:290] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 2 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:33.971827 19334 consensus_queue.cc:260] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.973017 19334 raft_consensus.cc:3060] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.977934 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8afd9e635b348b0938920cd55a0615a" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:33.978283 19538 consensus_queue.cc:1048] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:33.978730 19409 raft_consensus.cc:3060] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.979174 19411 raft_consensus.cc:515] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.980487 19408 raft_consensus.cc:1240] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Rejecting Update request from peer 2a776fc68e534b9ea0f748905e1de9e8 for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:33.983363 19334 raft_consensus.cc:2468] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 2.
I20260709 13:06:33.984992 19449 leader_election.cc:304] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:33.986099 19538 raft_consensus.cc:2804] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:33.987149 19586 consensus_queue.cc:1059] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:33.988327 19586 raft_consensus.cc:3055] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:33.988741 19586 raft_consensus.cc:740] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:33.986944 19538 raft_consensus.cc:697] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:33.989707 19586 consensus_queue.cc:260] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.990069 19538 consensus_queue.cc:237] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:33.990919 19586 raft_consensus.cc:3060] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.991916 19409 raft_consensus.cc:2468] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 2.
I20260709 13:06:33.999104 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:34.003021 19411 leader_election.cc:290] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:34.002384 19234 catalog_manager.cc:5697] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 1 to 2, leader changed from 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65) to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:34.004927 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edbcff77f1411481183330d339edd8" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:34.012985 19482 raft_consensus.cc:2468] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 2.
I20260709 13:06:34.014683 19376 leader_election.cc:304] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:34.015643 19571 raft_consensus.cc:2804] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:34.016291 19571 raft_consensus.cc:697] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:34.017380 19571 consensus_queue.cc:237] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:34.021653 19586 consensus_queue.cc:1048] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:34.026063 19232 catalog_manager.cc:5697] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 1 to 2, leader changed from 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67) to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.030236 19538 raft_consensus.cc:993] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8: : Instructing follower 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:34.030879 19583 raft_consensus.cc:1081] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Signalling peer 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:34.035730 19335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:57856
I20260709 13:06:34.036552 19335 raft_consensus.cc:493] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.037211 19335 raft_consensus.cc:3060] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.040143 19583 consensus_queue.cc:1048] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:34.046537 19335 raft_consensus.cc:515] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:34.050300 19335 leader_election.cc:290] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:34.051033 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:34.051745 19411 raft_consensus.cc:3060] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.051545 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468cc51b6ba9441d8a566322b6e6bcbc" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:34.052345 19482 raft_consensus.cc:3055] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:34.052742 19482 raft_consensus.cc:740] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:34.053633 19482 consensus_queue.cc:260] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:34.054802 19482 raft_consensus.cc:3060] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.060026 19411 raft_consensus.cc:2468] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:34.061455 19300 leader_election.cc:304] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:34.061467 19482 raft_consensus.cc:2468] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:34.062299 19531 raft_consensus.cc:2804] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:34.062773 19531 raft_consensus.cc:697] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:34.063649 19531 consensus_queue.cc:237] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:34.071267 19234 catalog_manager.cc:5697] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 1 to 2, leader changed from 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67) to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.120322 19538 consensus_queue.cc:1048] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:34.128414 19583 raft_consensus.cc:993] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8: : Instructing follower 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:34.129046 19583 raft_consensus.cc:1081] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Signalling peer 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:34.131184 19334 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:57856
I20260709 13:06:34.133622 19334 raft_consensus.cc:493] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.134078 19334 raft_consensus.cc:3060] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.135221 19583 consensus_queue.cc:1048] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:34.142016 19334 raft_consensus.cc:515] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:34.148113 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:34.148397 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc4f89a87a274604883f9b4f91d9872b" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:34.148835 19411 raft_consensus.cc:3060] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.149169 19482 raft_consensus.cc:3055] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:34.149248 19334 leader_election.cc:290] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:34.149703 19482 raft_consensus.cc:740] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:34.150673 19482 consensus_queue.cc:260] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:34.151597 19482 raft_consensus.cc:3060] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.155257 19411 raft_consensus.cc:2468] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:34.156533 19300 leader_election.cc:304] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:34.157300 19531 raft_consensus.cc:2804] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:34.157598 19482 raft_consensus.cc:2468] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:34.157823 19531 raft_consensus.cc:697] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:34.158885 19531 consensus_queue.cc:237] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:34.166925 19232 catalog_manager.cc:5697] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 1 to 2, leader changed from 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67) to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.260727 19334 raft_consensus.cc:1275] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.262080 19571 consensus_queue.cc:1048] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.270053 19482 raft_consensus.cc:1275] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.272439 19577 consensus_queue.cc:1048] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.300885 19411 raft_consensus.cc:1275] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.301800 19482 raft_consensus.cc:1275] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.302387 19531 consensus_queue.cc:1048] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.303393 19563 consensus_queue.cc:1048] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.340790 19411 raft_consensus.cc:1275] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.342324 19531 consensus_queue.cc:1048] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.347266 19482 raft_consensus.cc:1275] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.348716 19571 consensus_queue.cc:1048] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.352662 19482 raft_consensus.cc:1275] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.354475 19556 consensus_queue.cc:1048] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.360261 19334 raft_consensus.cc:1275] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.362721 19571 consensus_queue.cc:1048] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.386655 19334 raft_consensus.cc:1275] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.388163 19583 consensus_queue.cc:1048] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.397089 19411 raft_consensus.cc:1275] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.398936 19538 consensus_queue.cc:1048] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.465041 19485 raft_consensus.cc:1275] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.466768 19571 consensus_queue.cc:1048] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.475409 19334 raft_consensus.cc:3060] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.483220 19334 raft_consensus.cc:1275] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.484632 19571 consensus_queue.cc:1048] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.514333 19411 raft_consensus.cc:1275] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.515775 19556 consensus_queue.cc:1048] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.524278 19485 raft_consensus.cc:1275] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.525909 19564 consensus_queue.cc:1048] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.628952 19411 raft_consensus.cc:1275] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.630535 19564 consensus_queue.cc:1048] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.639034 19485 raft_consensus.cc:1275] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.640918 19556 consensus_queue.cc:1048] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:36.847167 19234 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58088:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:36.850533 19234 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:36.912946 19321 tablet_service.cc:1511] Processing CreateTablet for tablet 95cb88079f21459cac0767e8a11ed708 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:36.915252 19321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cb88079f21459cac0767e8a11ed708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.914990 19393 tablet_service.cc:1511] Processing CreateTablet for tablet 95cb88079f21459cac0767e8a11ed708 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:36.916270 19475 tablet_service.cc:1511] Processing CreateTablet for tablet 95cb88079f21459cac0767e8a11ed708 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:36.916863 19393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cb88079f21459cac0767e8a11ed708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.918188 19475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cb88079f21459cac0767e8a11ed708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.922775 19472 tablet_service.cc:1511] Processing CreateTablet for tablet 5b68875e4cde4edf9c9840b172043454 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:36.923208 19399 tablet_service.cc:1511] Processing CreateTablet for tablet 5b68875e4cde4edf9c9840b172043454 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:36.924585 19472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b68875e4cde4edf9c9840b172043454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.924585 19399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b68875e4cde4edf9c9840b172043454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.926352 19394 tablet_service.cc:1511] Processing CreateTablet for tablet a30a82466431423e9597c9dee074bcf1 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:36.928267 19394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30a82466431423e9597c9dee074bcf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.932163 19324 tablet_service.cc:1511] Processing CreateTablet for tablet a30a82466431423e9597c9dee074bcf1 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:36.932590 19325 tablet_service.cc:1511] Processing CreateTablet for tablet 5b68875e4cde4edf9c9840b172043454 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:36.932722 19397 tablet_service.cc:1511] Processing CreateTablet for tablet b843b9716ea0488b94a41b935f80f794 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:36.934345 19397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b843b9716ea0488b94a41b935f80f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.934583 19323 tablet_service.cc:1511] Processing CreateTablet for tablet b843b9716ea0488b94a41b935f80f794 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:36.935470 19325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b68875e4cde4edf9c9840b172043454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.936738 19324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30a82466431423e9597c9dee074bcf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.937808 19320 tablet_service.cc:1511] Processing CreateTablet for tablet 33748250b0014ce08aa43709d62646fc (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:36.938495 19323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b843b9716ea0488b94a41b935f80f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.939327 19320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33748250b0014ce08aa43709d62646fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.946784 19473 tablet_service.cc:1511] Processing CreateTablet for tablet b843b9716ea0488b94a41b935f80f794 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:36.947240 19318 tablet_service.cc:1511] Processing CreateTablet for tablet 42afcf5267814a8a97dc8562869ee54f (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:36.948421 19473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b843b9716ea0488b94a41b935f80f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.948954 19318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42afcf5267814a8a97dc8562869ee54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.950711 19468 tablet_service.cc:1511] Processing CreateTablet for tablet 33748250b0014ce08aa43709d62646fc (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:36.951296 19326 tablet_service.cc:1511] Processing CreateTablet for tablet 2efe92bdb3104343a4545e1b015c9fce (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:36.952477 19468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33748250b0014ce08aa43709d62646fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.953162 19326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe92bdb3104343a4545e1b015c9fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.945155 19467 tablet_service.cc:1511] Processing CreateTablet for tablet a30a82466431423e9597c9dee074bcf1 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:36.955408 19467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30a82466431423e9597c9dee074bcf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.957139 19319 tablet_service.cc:1511] Processing CreateTablet for tablet 1a5b17c161694271849e71bb7476cda3 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:36.959115 19319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5b17c161694271849e71bb7476cda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.968554 19395 tablet_service.cc:1511] Processing CreateTablet for tablet 33748250b0014ce08aa43709d62646fc (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:36.973389 19395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33748250b0014ce08aa43709d62646fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.973251 19396 tablet_service.cc:1511] Processing CreateTablet for tablet 2efe92bdb3104343a4545e1b015c9fce (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:36.972867 19400 tablet_service.cc:1511] Processing CreateTablet for tablet 42afcf5267814a8a97dc8562869ee54f (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:36.974838 19396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe92bdb3104343a4545e1b015c9fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.975507 19400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42afcf5267814a8a97dc8562869ee54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.976177 19474 tablet_service.cc:1511] Processing CreateTablet for tablet 42afcf5267814a8a97dc8562869ee54f (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:36.977805 19474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42afcf5267814a8a97dc8562869ee54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.984128 19471 tablet_service.cc:1511] Processing CreateTablet for tablet 2efe92bdb3104343a4545e1b015c9fce (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:36.984688 19469 tablet_service.cc:1511] Processing CreateTablet for tablet 1a5b17c161694271849e71bb7476cda3 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:36.985842 19471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe92bdb3104343a4545e1b015c9fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.986593 19469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5b17c161694271849e71bb7476cda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.986505 19322 tablet_service.cc:1511] Processing CreateTablet for tablet 1e0ce4fd1f904327b43092aff73b5ce8 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:36.988070 19322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0ce4fd1f904327b43092aff73b5ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:36.996050 19401 tablet_service.cc:1511] Processing CreateTablet for tablet 1a5b17c161694271849e71bb7476cda3 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
W20260709 13:06:36.997745 19282 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:36.999159 19398 tablet_service.cc:1511] Processing CreateTablet for tablet 1e0ce4fd1f904327b43092aff73b5ce8 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:37.000797 19401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5b17c161694271849e71bb7476cda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:37.005065 19470 tablet_service.cc:1511] Processing CreateTablet for tablet 1e0ce4fd1f904327b43092aff73b5ce8 (DEFAULT_TABLE table=soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:37.005846 19398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0ce4fd1f904327b43092aff73b5ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:37.006654 19470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0ce4fd1f904327b43092aff73b5ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:37.144846 19625 tablet_bootstrap.cc:492] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.185595 19625 tablet_bootstrap.cc:654] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.196137 19624 tablet_bootstrap.cc:492] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.196743 19627 tablet_bootstrap.cc:492] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.207700 19624 tablet_bootstrap.cc:654] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.210142 19625 tablet_bootstrap.cc:492] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.210547 19627 tablet_bootstrap.cc:654] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.210737 19625 ts_tablet_manager.cc:1403] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.066s	user 0.010s	sys 0.009s
I20260709 13:06:37.213773 19624 tablet_bootstrap.cc:492] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.214445 19624 ts_tablet_manager.cc:1403] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.000s
I20260709 13:06:37.215305 19625 raft_consensus.cc:359] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.216377 19625 raft_consensus.cc:385] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.216667 19625 raft_consensus.cc:740] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.217386 19625 consensus_queue.cc:260] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.218154 19627 tablet_bootstrap.cc:492] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.217859 19624 raft_consensus.cc:359] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.218652 19624 raft_consensus.cc:385] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.218686 19627 ts_tablet_manager.cc:1403] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.004s
I20260709 13:06:37.219130 19624 raft_consensus.cc:740] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.220358 19625 ts_tablet_manager.cc:1434] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:37.219991 19624 consensus_queue.cc:260] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.221601 19625 tablet_bootstrap.cc:492] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.222528 19627 raft_consensus.cc:359] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.223284 19627 raft_consensus.cc:385] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.223573 19627 raft_consensus.cc:740] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.224524 19627 consensus_queue.cc:260] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.228840 19624 ts_tablet_manager.cc:1434] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.014s	user 0.000s	sys 0.006s
I20260709 13:06:37.229593 19625 tablet_bootstrap.cc:654] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.230109 19624 tablet_bootstrap.cc:492] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.233841 19627 ts_tablet_manager.cc:1434] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:06:37.235131 19627 tablet_bootstrap.cc:492] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.238951 19624 tablet_bootstrap.cc:654] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.242924 19627 tablet_bootstrap.cc:654] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.264099 19624 tablet_bootstrap.cc:492] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.264700 19624 ts_tablet_manager.cc:1403] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.035s	user 0.019s	sys 0.005s
I20260709 13:06:37.264978 19625 tablet_bootstrap.cc:492] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.265643 19625 ts_tablet_manager.cc:1403] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.044s	user 0.024s	sys 0.019s
I20260709 13:06:37.267735 19627 tablet_bootstrap.cc:492] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.268278 19627 ts_tablet_manager.cc:1403] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.000s
I20260709 13:06:37.267998 19624 raft_consensus.cc:359] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.268824 19624 raft_consensus.cc:385] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.268821 19625 raft_consensus.cc:359] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.269464 19624 raft_consensus.cc:740] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.269840 19625 raft_consensus.cc:385] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.270169 19625 raft_consensus.cc:740] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.270378 19624 consensus_queue.cc:260] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.270951 19625 consensus_queue.cc:260] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.271095 19627 raft_consensus.cc:359] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.273011 19624 ts_tablet_manager.cc:1434] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 13:06:37.273644 19625 ts_tablet_manager.cc:1434] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:37.274088 19624 tablet_bootstrap.cc:492] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.274636 19625 tablet_bootstrap.cc:492] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.271880 19627 raft_consensus.cc:385] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.275004 19627 raft_consensus.cc:740] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.275883 19627 consensus_queue.cc:260] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.278276 19627 ts_tablet_manager.cc:1434] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:06:37.279256 19627 tablet_bootstrap.cc:492] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.281935 19624 tablet_bootstrap.cc:654] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.281927 19625 tablet_bootstrap.cc:654] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.286389 19627 tablet_bootstrap.cc:654] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:37.327998 19375 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:37185, user_credentials={real_user=slave}} blocked reactor thread for 44161.4us
I20260709 13:06:37.343240 19626 raft_consensus.cc:493] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.344106 19626 raft_consensus.cc:515] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.349443 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
W20260709 13:06:37.352082 19300 leader_election.cc:343] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.353628 19626 leader_election.cc:290] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.355440 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
W20260709 13:06:37.358151 19301 leader_election.cc:343] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.358822 19301 leader_election.cc:304] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05; no voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:37.365631 19626 raft_consensus.cc:2749] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:37.391755 19624 tablet_bootstrap.cc:492] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.392630 19624 ts_tablet_manager.cc:1403] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.119s	user 0.056s	sys 0.035s
I20260709 13:06:37.396359 19624 raft_consensus.cc:359] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.397127 19624 raft_consensus.cc:385] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.397559 19624 raft_consensus.cc:740] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.397724 19625 tablet_bootstrap.cc:492] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.397965 19627 tablet_bootstrap.cc:492] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.398314 19625 ts_tablet_manager.cc:1403] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.124s	user 0.062s	sys 0.051s
I20260709 13:06:37.398517 19627 ts_tablet_manager.cc:1403] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.119s	user 0.041s	sys 0.074s
I20260709 13:06:37.398401 19624 consensus_queue.cc:260] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.401105 19624 ts_tablet_manager.cc:1434] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 13:06:37.402192 19624 tablet_bootstrap.cc:492] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.401919 19625 raft_consensus.cc:359] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.402714 19625 raft_consensus.cc:385] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.402220 19627 raft_consensus.cc:359] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.403083 19625 raft_consensus.cc:740] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.403417 19627 raft_consensus.cc:385] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.403834 19627 raft_consensus.cc:740] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.404798 19627 consensus_queue.cc:260] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.403905 19625 consensus_queue.cc:260] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.410029 19624 tablet_bootstrap.cc:654] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.410511 19627 ts_tablet_manager.cc:1434] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:06:37.411656 19627 tablet_bootstrap.cc:492] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.414808 19625 ts_tablet_manager.cc:1434] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.003s
I20260709 13:06:37.416253 19625 tablet_bootstrap.cc:492] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.419114 19627 tablet_bootstrap.cc:654] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.422145 19624 tablet_bootstrap.cc:492] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.422698 19624 ts_tablet_manager.cc:1403] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.007s
I20260709 13:06:37.424181 19625 tablet_bootstrap.cc:654] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.425827 19624 raft_consensus.cc:359] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.426644 19624 raft_consensus.cc:385] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.426942 19624 raft_consensus.cc:740] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.428301 19624 consensus_queue.cc:260] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.431476 19627 tablet_bootstrap.cc:492] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.431699 19624 ts_tablet_manager.cc:1434] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 13:06:37.432344 19627 ts_tablet_manager.cc:1403] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.005s
I20260709 13:06:37.433219 19624 tablet_bootstrap.cc:492] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.436203 19625 tablet_bootstrap.cc:492] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.435972 19627 raft_consensus.cc:359] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.436794 19625 ts_tablet_manager.cc:1403] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260709 13:06:37.437109 19627 raft_consensus.cc:385] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.437619 19627 raft_consensus.cc:740] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.438588 19627 consensus_queue.cc:260] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.440505 19625 raft_consensus.cc:359] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.441177 19624 tablet_bootstrap.cc:654] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.441371 19625 raft_consensus.cc:385] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.441773 19625 raft_consensus.cc:740] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.442610 19625 consensus_queue.cc:260] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.444005 19627 ts_tablet_manager.cc:1434] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.004s
I20260709 13:06:37.445264 19627 tablet_bootstrap.cc:492] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.445843 19625 ts_tablet_manager.cc:1434] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 13:06:37.447309 19625 tablet_bootstrap.cc:492] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.447755 19613 raft_consensus.cc:493] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.448364 19613 raft_consensus.cc:515] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.451297 19613 leader_election.cc:290] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.453984 19627 tablet_bootstrap.cc:654] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.454571 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2efe92bdb3104343a4545e1b015c9fce" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:37.455345 19482 raft_consensus.cc:2468] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:37.456856 19376 leader_election.cc:304] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.458999 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2efe92bdb3104343a4545e1b015c9fce" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:37.459702 19334 raft_consensus.cc:2468] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:37.460963 19613 raft_consensus.cc:2804] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:37.461932 19613 raft_consensus.cc:493] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:37.462286 19613 raft_consensus.cc:3060] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.464214 19625 tablet_bootstrap.cc:654] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.478117 19624 tablet_bootstrap.cc:492] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.478895 19624 ts_tablet_manager.cc:1403] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.046s	user 0.017s	sys 0.014s
I20260709 13:06:37.483282 19624 raft_consensus.cc:359] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.484393 19624 raft_consensus.cc:385] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.484954 19624 raft_consensus.cc:740] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.486320 19624 consensus_queue.cc:260] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.488988 19626 raft_consensus.cc:493] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.489784 19626 raft_consensus.cc:515] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.492273 19626 leader_election.cc:290] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.493894 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
W20260709 13:06:37.495752 19300 leader_election.cc:343] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.496935 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:37.499187 19624 ts_tablet_manager.cc:1434] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.020s	user 0.011s	sys 0.000s
W20260709 13:06:37.499666 19301 leader_election.cc:343] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.500371 19301 leader_election.cc:304] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05; no voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:37.500778 19624 tablet_bootstrap.cc:492] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.501480 19626 raft_consensus.cc:2749] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:37.504125 19609 raft_consensus.cc:493] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.504931 19609 raft_consensus.cc:515] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.512080 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:37.512346 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:37.513327 19409 raft_consensus.cc:2468] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:37.517179 19624 tablet_bootstrap.cc:654] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.519069 19449 leader_election.cc:304] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.509099 19609 leader_election.cc:290] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:37.523465 19609 raft_consensus.cc:2804] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:37.524493 19645 raft_consensus.cc:493] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.525182 19609 raft_consensus.cc:493] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:37.526665 19609 raft_consensus.cc:3060] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.525149 19645 raft_consensus.cc:515] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.528515 19334 raft_consensus.cc:2468] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:37.531551 19645 leader_election.cc:290] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:37.532284 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:37.533169 19335 raft_consensus.cc:2468] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:37.533106 19624 tablet_bootstrap.cc:492] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.535472 19624 ts_tablet_manager.cc:1403] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.035s	user 0.019s	sys 0.005s
I20260709 13:06:37.535382 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:37.536445 19409 raft_consensus.cc:2468] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:37.540040 19624 raft_consensus.cc:359] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.541144 19624 raft_consensus.cc:385] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.541553 19624 raft_consensus.cc:740] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.542771 19624 consensus_queue.cc:260] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.544724 19449 leader_election.cc:304] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.546849 19624 ts_tablet_manager.cc:1434] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 13:06:37.548507 19645 raft_consensus.cc:2804] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:37.549019 19645 raft_consensus.cc:493] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:37.549381 19645 raft_consensus.cc:3060] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.560243 19624 tablet_bootstrap.cc:492] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.566461 19645 raft_consensus.cc:515] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.568744 19645 leader_election.cc:290] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:37.572867 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:37.582716 19334 raft_consensus.cc:3060] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.574654 19645 raft_consensus.cc:493] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.585196 19609 raft_consensus.cc:515] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.586055 19645 raft_consensus.cc:515] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.588168 19645 leader_election.cc:290] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:37.576633 19624 tablet_bootstrap.cc:654] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:37.579869 19375 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:37185, user_credentials={real_user=slave}} blocked reactor thread for 93396.7us
I20260709 13:06:37.570909 19613 raft_consensus.cc:515] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.592185 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:37.592413 19613 leader_election.cc:290] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.593034 19335 raft_consensus.cc:3060] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.600553 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2efe92bdb3104343a4545e1b015c9fce" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:37.594930 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:37.574337 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:37.602118 19625 tablet_bootstrap.cc:492] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.602597 19409 raft_consensus.cc:3060] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.602934 19625 ts_tablet_manager.cc:1403] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.156s	user 0.069s	sys 0.070s
I20260709 13:06:37.597433 19334 raft_consensus.cc:2468] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:37.597293 19336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:37.602118 19627 tablet_bootstrap.cc:492] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
W20260709 13:06:37.602785 19449 leader_election.cc:343] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.604804 19627 ts_tablet_manager.cc:1403] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.160s	user 0.028s	sys 0.065s
I20260709 13:06:37.601274 19482 raft_consensus.cc:3060] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.606680 19625 raft_consensus.cc:359] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.607457 19625 raft_consensus.cc:385] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.607753 19625 raft_consensus.cc:740] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.607892 19335 raft_consensus.cc:2468] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:37.607921 19627 raft_consensus.cc:359] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.608971 19627 raft_consensus.cc:385] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.608559 19625 consensus_queue.cc:260] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.609292 19627 raft_consensus.cc:740] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
W20260709 13:06:37.610597 19449 leader_election.cc:343] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.611085 19449 leader_election.cc:304] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8; no voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:37.611248 19625 ts_tablet_manager.cc:1434] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 13:06:37.611099 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:37.612012 19411 raft_consensus.cc:3060] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.612382 19645 raft_consensus.cc:2749] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:37.612618 19625 tablet_bootstrap.cc:492] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.612382 19449 leader_election.cc:304] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:37.614207 19449 leader_election.cc:304] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:37.615110 19646 raft_consensus.cc:2804] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:37.616389 19645 raft_consensus.cc:2804] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:37.618536 19627 consensus_queue.cc:260] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.622219 19411 raft_consensus.cc:2468] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:37.625977 19625 tablet_bootstrap.cc:654] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.627698 19627 ts_tablet_manager.cc:1434] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.023s	user 0.001s	sys 0.007s
I20260709 13:06:37.628261 19409 raft_consensus.cc:2468] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:37.597790 19624 tablet_bootstrap.cc:492] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.629158 19627 tablet_bootstrap.cc:492] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.624349 19482 raft_consensus.cc:2468] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:37.629921 19624 ts_tablet_manager.cc:1403] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.082s	user 0.010s	sys 0.007s
I20260709 13:06:37.631062 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2efe92bdb3104343a4545e1b015c9fce" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:37.631780 19335 raft_consensus.cc:3060] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.634227 19624 raft_consensus.cc:359] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.635136 19624 raft_consensus.cc:385] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.635535 19624 raft_consensus.cc:740] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.637576 19376 leader_election.cc:304] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.638531 19613 raft_consensus.cc:2804] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:37.639161 19613 raft_consensus.cc:697] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:37.636677 19624 consensus_queue.cc:260] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.640061 19613 consensus_queue.cc:237] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.643934 19335 raft_consensus.cc:2468] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:37.647118 19627 tablet_bootstrap.cc:654] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.659986 19624 ts_tablet_manager.cc:1434] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.001s
I20260709 13:06:37.660290 19609 leader_election.cc:290] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:37.661437 19624 tablet_bootstrap.cc:492] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.661898 19634 raft_consensus.cc:493] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.662724 19634 raft_consensus.cc:515] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.668061 19634 leader_election.cc:290] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.671296 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a5b17c161694271849e71bb7476cda3" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:37.672654 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a5b17c161694271849e71bb7476cda3" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
W20260709 13:06:37.673171 19376 leader_election.cc:343] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.673759 19335 raft_consensus.cc:2468] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:37.675108 19375 leader_election.cc:304] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:37.676457 19634 raft_consensus.cc:2804] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:37.676935 19634 raft_consensus.cc:493] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:37.677371 19634 raft_consensus.cc:3060] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.680806 19624 tablet_bootstrap.cc:654] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.688282 19234 catalog_manager.cc:5697] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:37.688648 19634 raft_consensus.cc:515] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.692212 19634 leader_election.cc:290] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.693624 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a5b17c161694271849e71bb7476cda3" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
W20260709 13:06:37.695989 19376 leader_election.cc:343] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:37.696043 19645 raft_consensus.cc:697] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:37.693590 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a5b17c161694271849e71bb7476cda3" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:37.698326 19335 raft_consensus.cc:3060] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.698902 19646 raft_consensus.cc:697] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:37.700454 19646 consensus_queue.cc:237] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.708302 19625 tablet_bootstrap.cc:492] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.709666 19625 ts_tablet_manager.cc:1403] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.097s	user 0.031s	sys 0.044s
I20260709 13:06:37.710934 19335 raft_consensus.cc:2468] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:37.712425 19375 leader_election.cc:304] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:37.713379 19634 raft_consensus.cc:2804] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:37.698416 19645 consensus_queue.cc:237] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.714115 19634 raft_consensus.cc:697] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:37.715315 19634 consensus_queue.cc:237] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.720646 19627 tablet_bootstrap.cc:492] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.721248 19627 ts_tablet_manager.cc:1403] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.092s	user 0.022s	sys 0.008s
I20260709 13:06:37.724241 19627 raft_consensus.cc:359] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.725106 19627 raft_consensus.cc:385] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.725453 19627 raft_consensus.cc:740] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.726244 19627 consensus_queue.cc:260] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.728858 19627 ts_tablet_manager.cc:1434] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 13:06:37.729981 19627 tablet_bootstrap.cc:492] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.730197 19625 raft_consensus.cc:359] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.731272 19625 raft_consensus.cc:385] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.731631 19625 raft_consensus.cc:740] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.731712 19230 catalog_manager.cc:5697] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:37.732719 19625 consensus_queue.cc:260] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.735659 19625 ts_tablet_manager.cc:1434] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 13:06:37.731766 19624 tablet_bootstrap.cc:492] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.737061 19625 tablet_bootstrap.cc:492] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.737185 19624 ts_tablet_manager.cc:1403] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.076s	user 0.017s	sys 0.012s
I20260709 13:06:37.743342 19627 tablet_bootstrap.cc:654] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.743284 19624 raft_consensus.cc:359] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.744390 19624 raft_consensus.cc:385] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.744760 19624 raft_consensus.cc:740] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.744920 19625 tablet_bootstrap.cc:654] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.745718 19624 consensus_queue.cc:260] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.748169 19624 ts_tablet_manager.cc:1434] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:06:37.749431 19624 tablet_bootstrap.cc:492] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05: Bootstrap starting.
I20260709 13:06:37.757007 19624 tablet_bootstrap.cc:654] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.766671 19627 tablet_bootstrap.cc:492] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.767369 19627 ts_tablet_manager.cc:1403] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.038s	user 0.017s	sys 0.010s
I20260709 13:06:37.770778 19624 tablet_bootstrap.cc:492] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05: No bootstrap required, opened a new log
I20260709 13:06:37.771385 19624 ts_tablet_manager.cc:1403] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.001s
I20260709 13:06:37.772223 19627 raft_consensus.cc:359] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.773061 19627 raft_consensus.cc:385] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.773373 19627 raft_consensus.cc:740] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.774256 19627 consensus_queue.cc:260] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.774360 19624 raft_consensus.cc:359] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.775125 19624 raft_consensus.cc:385] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.775395 19624 raft_consensus.cc:740] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.776058 19624 consensus_queue.cc:260] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.776892 19627 ts_tablet_manager.cc:1434] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.003s
I20260709 13:06:37.778049 19627 tablet_bootstrap.cc:492] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.781594 19625 tablet_bootstrap.cc:492] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.782240 19625 ts_tablet_manager.cc:1403] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.045s	user 0.012s	sys 0.015s
I20260709 13:06:37.781240 19234 catalog_manager.cc:5697] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:37.785727 19625 raft_consensus.cc:359] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.786284 19627 tablet_bootstrap.cc:654] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.786515 19625 raft_consensus.cc:385] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.786775 19624 ts_tablet_manager.cc:1434] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:06:37.786926 19625 raft_consensus.cc:740] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.787941 19625 consensus_queue.cc:260] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.790485 19625 ts_tablet_manager.cc:1434] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:37.791577 19625 tablet_bootstrap.cc:492] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.798256 19625 tablet_bootstrap.cc:654] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.806375 19233 catalog_manager.cc:5697] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:37.809527 19625 tablet_bootstrap.cc:492] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.810102 19625 ts_tablet_manager.cc:1403] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.007s
I20260709 13:06:37.814277 19625 raft_consensus.cc:359] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.815042 19625 raft_consensus.cc:385] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.815320 19625 raft_consensus.cc:740] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.816598 19625 consensus_queue.cc:260] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.819198 19625 ts_tablet_manager.cc:1434] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:06:37.820225 19625 tablet_bootstrap.cc:492] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8: Bootstrap starting.
I20260709 13:06:37.824951 19625 tablet_bootstrap.cc:654] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.827961 19627 tablet_bootstrap.cc:492] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.828702 19627 ts_tablet_manager.cc:1403] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.051s	user 0.008s	sys 0.033s
I20260709 13:06:37.830175 19625 tablet_bootstrap.cc:492] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8: No bootstrap required, opened a new log
I20260709 13:06:37.830711 19625 ts_tablet_manager.cc:1403] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:06:37.831202 19627 raft_consensus.cc:359] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.832002 19627 raft_consensus.cc:385] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.832288 19627 raft_consensus.cc:740] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.833055 19627 consensus_queue.cc:260] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.833465 19625 raft_consensus.cc:359] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.834133 19625 raft_consensus.cc:385] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.834427 19625 raft_consensus.cc:740] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.835073 19625 consensus_queue.cc:260] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.837111 19625 ts_tablet_manager.cc:1434] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:06:37.837936 19627 ts_tablet_manager.cc:1434] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 13:06:37.839093 19627 tablet_bootstrap.cc:492] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28: Bootstrap starting.
I20260709 13:06:37.841919 19626 raft_consensus.cc:493] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.842517 19626 raft_consensus.cc:515] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.844391 19626 leader_election.cc:290] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.845693 19627 tablet_bootstrap.cc:654] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:37.846060 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:37.846223 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:37.846859 19485 raft_consensus.cc:2468] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 0.
I20260709 13:06:37.846908 19409 raft_consensus.cc:2468] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 0.
I20260709 13:06:37.848595 19300 leader_election.cc:304] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.850000 19626 raft_consensus.cc:2804] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:37.850373 19626 raft_consensus.cc:493] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:37.850699 19626 raft_consensus.cc:3060] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.856171 19627 tablet_bootstrap.cc:492] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28: No bootstrap required, opened a new log
I20260709 13:06:37.856784 19627 ts_tablet_manager.cc:1403] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.011s
I20260709 13:06:37.858857 19626 raft_consensus.cc:515] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.859057 19627 raft_consensus.cc:359] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.859601 19627 raft_consensus.cc:385] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:37.859834 19627 raft_consensus.cc:740] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Initialized, Role: FOLLOWER
I20260709 13:06:37.860455 19627 consensus_queue.cc:260] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.860824 19626 leader_election.cc:290] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.861666 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:37.862252 19409 raft_consensus.cc:3060] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.862236 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:37.862900 19627 ts_tablet_manager.cc:1434] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.005s
I20260709 13:06:37.863098 19485 raft_consensus.cc:3060] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.869599 19409 raft_consensus.cc:2468] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 1.
I20260709 13:06:37.869614 19485 raft_consensus.cc:2468] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 1.
I20260709 13:06:37.871210 19300 leader_election.cc:304] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.872138 19626 raft_consensus.cc:2804] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:37.872776 19626 raft_consensus.cc:697] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:37.873756 19626 consensus_queue.cc:237] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.880771 19634 raft_consensus.cc:493] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.881389 19634 raft_consensus.cc:515] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.884690 19634 leader_election.cc:290] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.884719 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:37.885838 19335 raft_consensus.cc:2468] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:37.885792 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:37.884603 19230 catalog_manager.cc:5697] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:37.886759 19485 raft_consensus.cc:2468] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:37.887409 19375 leader_election.cc:304] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.888223 19634 raft_consensus.cc:2804] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:37.888672 19634 raft_consensus.cc:493] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:37.889010 19634 raft_consensus.cc:3060] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.896023 19613 raft_consensus.cc:493] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:37.896163 19634 raft_consensus.cc:515] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.896737 19613 raft_consensus.cc:515] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.898633 19613 leader_election.cc:290] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.898654 19634 leader_election.cc:290] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.900008 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33748250b0014ce08aa43709d62646fc" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:37.900496 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:37.900713 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:37.901003 19485 raft_consensus.cc:2468] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:37.901153 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33748250b0014ce08aa43709d62646fc" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:37.901382 19335 raft_consensus.cc:3060] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.901858 19334 raft_consensus.cc:2468] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:37.902946 19375 leader_election.cc:304] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.901186 19482 raft_consensus.cc:3060] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.904000 19634 raft_consensus.cc:2804] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:37.904366 19634 raft_consensus.cc:493] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:37.904774 19634 raft_consensus.cc:3060] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.909149 19335 raft_consensus.cc:2468] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:37.910502 19375 leader_election.cc:304] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.911113 19482 raft_consensus.cc:2468] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:37.911438 19613 raft_consensus.cc:2804] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:37.912014 19613 raft_consensus.cc:697] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:37.912869 19613 consensus_queue.cc:237] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.914268 19634 raft_consensus.cc:515] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.917191 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33748250b0014ce08aa43709d62646fc" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:37.917325 19634 leader_election.cc:290] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:37.917554 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33748250b0014ce08aa43709d62646fc" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:37.917841 19334 raft_consensus.cc:3060] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.918406 19482 raft_consensus.cc:3060] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:37.921985 19230 catalog_manager.cc:5697] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:37.924741 19482 raft_consensus.cc:2468] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:37.926527 19376 leader_election.cc:304] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:37.927434 19613 raft_consensus.cc:2804] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:37.927874 19613 raft_consensus.cc:697] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:37.928719 19613 consensus_queue.cc:237] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:37.930114 19334 raft_consensus.cc:2468] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:37.938588 19230 catalog_manager.cc:5697] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:38.001842 19282 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:38.086081 19634 raft_consensus.cc:493] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:38.086750 19634 raft_consensus.cc:515] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.088510 19634 leader_election.cc:290] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:38.089572 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:38.089564 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8" is_pre_election: true
I20260709 13:06:38.090271 19334 raft_consensus.cc:2468] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:38.090456 19482 raft_consensus.cc:2468] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 0.
I20260709 13:06:38.091266 19375 leader_election.cc:304] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:38.092011 19634 raft_consensus.cc:2804] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:38.092279 19634 raft_consensus.cc:493] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:38.092530 19634 raft_consensus.cc:3060] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:38.098625 19634 raft_consensus.cc:515] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.100600 19634 leader_election.cc:290] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:38.101573 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:38.101773 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:38.102286 19334 raft_consensus.cc:3060] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:38.102312 19482 raft_consensus.cc:3060] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:38.109239 19482 raft_consensus.cc:2468] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:38.109239 19334 raft_consensus.cc:2468] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 1.
I20260709 13:06:38.110497 19375 leader_election.cc:304] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:38.111302 19634 raft_consensus.cc:2804] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:38.111742 19634 raft_consensus.cc:697] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:38.112221 19646 consensus_queue.cc:1048] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.112474 19634 consensus_queue.cc:237] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.121713 19650 consensus_queue.cc:1048] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:38.124205 19650 raft_consensus.cc:493] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:38.124794 19650 raft_consensus.cc:515] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.126314 19230 catalog_manager.cc:5697] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:38.127187 19650 leader_election.cc:290] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:38.137303 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a5b17c161694271849e71bb7476cda3" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:38.138262 19335 raft_consensus.cc:2393] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a776fc68e534b9ea0f748905e1de9e8 in current term 1: Already voted for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in this term.
I20260709 13:06:38.141626 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a5b17c161694271849e71bb7476cda3" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:38.145661 19449 leader_election.cc:304] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8; no voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:38.147706 19651 raft_consensus.cc:3060] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:38.148375 19650 consensus_queue.cc:1048] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.149066 19609 raft_consensus.cc:493] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:38.149675 19609 raft_consensus.cc:515] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.152014 19609 leader_election.cc:290] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:38.154096 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" is_pre_election: true
I20260709 13:06:38.154946 19411 raft_consensus.cc:2468] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:38.156525 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05" is_pre_election: true
I20260709 13:06:38.157384 19335 raft_consensus.cc:2468] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 0.
I20260709 13:06:38.157668 19449 leader_election.cc:304] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:38.159829 19609 raft_consensus.cc:2804] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:38.159972 19651 raft_consensus.cc:2749] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:38.160367 19609 raft_consensus.cc:493] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:38.161312 19609 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:38.174093 19646 consensus_queue.cc:1048] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:38.174401 19609 raft_consensus.cc:515] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.177032 19609 leader_election.cc:290] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:38.179761 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:38.180905 19334 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:38.181375 19411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:38.182101 19411 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:38.183712 19634 consensus_queue.cc:1048] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.187965 19411 raft_consensus.cc:2468] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:38.188716 19334 raft_consensus.cc:2468] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 1.
I20260709 13:06:38.189564 19449 leader_election.cc:304] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:38.190555 19609 raft_consensus.cc:2804] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:38.191083 19609 raft_consensus.cc:697] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:38.191944 19609 consensus_queue.cc:237] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.197233 19644 consensus_queue.cc:1048] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:38.226188 19230 catalog_manager.cc:5697] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 1 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:38.262985 19556 consensus_queue.cc:1048] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:38.273201 19634 consensus_queue.cc:1048] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.276170 19626 consensus_queue.cc:1048] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.288951 19644 consensus_queue.cc:1048] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.369200 19613 consensus_queue.cc:1048] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.379606 19634 consensus_queue.cc:1048] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.436478 19634 consensus_queue.cc:1048] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.447942 19613 consensus_queue.cc:1048] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.479624 19482 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
mode: GRACEFUL
new_leader_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:34882
I20260709 13:06:38.480188 19482 raft_consensus.cc:606] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Received request to transfer leadership to TS 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:38.485257 19482 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
mode: GRACEFUL
new_leader_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:34882
I20260709 13:06:38.485787 19482 raft_consensus.cc:606] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Received request to transfer leadership to TS 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:38.493484 19409 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
mode: GRACEFUL
new_leader_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:53720
I20260709 13:06:38.494082 19409 raft_consensus.cc:606] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Received request to transfer leadership to TS 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:38.501462 19482 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
mode: GRACEFUL
new_leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:34882
I20260709 13:06:38.501972 19482 raft_consensus.cc:606] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Received request to transfer leadership to TS c9592e56d9fa4405a1cdca7ccc6a4e28
I20260709 13:06:38.512365 19409 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
mode: GRACEFUL
new_leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:53720
I20260709 13:06:38.512889 19409 raft_consensus.cc:606] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Received request to transfer leadership to TS 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:38.573680 19646 raft_consensus.cc:993] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: : Instructing follower 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:38.574155 19646 raft_consensus.cc:1081] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Signalling peer 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:38.575707 19335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:57856
I20260709 13:06:38.576213 19335 raft_consensus.cc:493] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:38.576570 19335 raft_consensus.cc:3060] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.581485 19335 raft_consensus.cc:515] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.583200 19335 leader_election.cc:290] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:38.584048 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:38.584352 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95cb88079f21459cac0767e8a11ed708" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:38.584637 19409 raft_consensus.cc:3060] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.584986 19482 raft_consensus.cc:3055] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:38.585263 19482 raft_consensus.cc:740] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:38.586037 19482 consensus_queue.cc:260] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.587016 19482 raft_consensus.cc:3060] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.590000 19409 raft_consensus.cc:2468] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:38.591310 19300 leader_election.cc:304] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:38.592199 19626 raft_consensus.cc:2804] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:38.592531 19482 raft_consensus.cc:2468] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:38.592684 19626 raft_consensus.cc:697] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:38.593667 19626 consensus_queue.cc:237] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.602058 19230 catalog_manager.cc:5697] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 1 to 2, leader changed from 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67) to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:38.615046 19646 consensus_queue.cc:1048] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.624018 19609 raft_consensus.cc:993] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: : Instructing follower c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:38.624935 19609 raft_consensus.cc:1081] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Signalling peer c9592e56d9fa4405a1cdca7ccc6a4e28 to start an election
I20260709 13:06:38.626657 19411 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
 from {username='slave'} at 127.0.0.1:53706
I20260709 13:06:38.627262 19411 raft_consensus.cc:493] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:38.627635 19411 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.632848 19646 consensus_queue.cc:1048] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.636657 19411 raft_consensus.cc:515] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.639048 19409 raft_consensus.cc:1240] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Rejecting Update request from peer 2a776fc68e534b9ea0f748905e1de9e8 for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:38.640924 19411 leader_election.cc:290] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:38.642129 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:38.642596 19334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:38.642767 19482 raft_consensus.cc:3055] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:38.643174 19482 raft_consensus.cc:740] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:38.643904 19482 consensus_queue.cc:260] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.644761 19482 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:38.646941 19646 consensus_queue.cc:1175] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 2a776fc68e534b9ea0f748905e1de9e8 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:06:38.651165 19482 raft_consensus.cc:2468] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9592e56d9fa4405a1cdca7ccc6a4e28 in term 2.
I20260709 13:06:38.652488 19376 leader_election.cc:304] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 10584038f61a4dd4a618aa4d75689a05
I20260709 13:06:38.653331 19613 raft_consensus.cc:2804] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:38.653883 19613 raft_consensus.cc:697] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Becoming Leader. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:38.654748 19613 consensus_queue.cc:237] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.664812 19230 catalog_manager.cc:5697] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 reported cstate change: term changed from 1 to 2, leader changed from 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67) to c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:38.700455 19650 raft_consensus.cc:993] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: : Instructing follower 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:38.700959 19646 raft_consensus.cc:1081] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Signalling peer 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:38.702459 19335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:57856
I20260709 13:06:38.702977 19335 raft_consensus.cc:493] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:38.703333 19335 raft_consensus.cc:3060] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.708810 19613 consensus_queue.cc:1048] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:38.709602 19335 raft_consensus.cc:515] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.711041 19613 consensus_queue.cc:1048] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.712399 19335 leader_election.cc:290] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:38.712610 19482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:38.713768 19482 raft_consensus.cc:3055] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:38.714208 19482 raft_consensus.cc:740] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:38.714105 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b68875e4cde4edf9c9840b172043454" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:38.715091 19409 raft_consensus.cc:3060] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.715474 19482 consensus_queue.cc:260] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.717006 19482 raft_consensus.cc:3060] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.717775 19644 raft_consensus.cc:993] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: : Instructing follower 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:38.718421 19634 raft_consensus.cc:1081] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Signalling peer 10584038f61a4dd4a618aa4d75689a05 to start an election
I20260709 13:06:38.721091 19335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
 from {username='slave'} at 127.0.0.1:57844
I20260709 13:06:38.721971 19335 raft_consensus.cc:493] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:38.722332 19335 raft_consensus.cc:3060] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.728449 19409 raft_consensus.cc:2468] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:38.732973 19300 leader_election.cc:304] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:38.733282 19335 raft_consensus.cc:515] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.734365 19334 raft_consensus.cc:1240] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Rejecting Update request from peer c9592e56d9fa4405a1cdca7ccc6a4e28 for earlier term 1. Current term is 2. Ops: []
I20260709 13:06:38.734375 19626 raft_consensus.cc:2804] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:38.735345 19626 raft_consensus.cc:697] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:38.736809 19613 consensus_queue.cc:1059] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:38.737725 19482 raft_consensus.cc:2468] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:38.737064 19626 consensus_queue.cc:237] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.739207 19335 leader_election.cc:290] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Requested vote from peers c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545), 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67:35297)
I20260709 13:06:38.740790 19485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
I20260709 13:06:38.741554 19485 raft_consensus.cc:3060] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.741772 19644 raft_consensus.cc:3055] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:38.742218 19644 raft_consensus.cc:740] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:38.742745 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b843b9716ea0488b94a41b935f80f794" candidate_uuid: "10584038f61a4dd4a618aa4d75689a05" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:38.743180 19644 consensus_queue.cc:260] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.744683 19644 raft_consensus.cc:3060] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.752446 19485 raft_consensus.cc:2468] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:38.756289 19301 leader_election.cc:304] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:38.757196 19626 raft_consensus.cc:2804] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:38.757807 19626 raft_consensus.cc:697] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Leader. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:38.758706 19626 consensus_queue.cc:237] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.761672 19233 catalog_manager.cc:5697] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 1 to 2, leader changed from 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67) to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:38.765333 19409 raft_consensus.cc:2468] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10584038f61a4dd4a618aa4d75689a05 in term 2.
I20260709 13:06:38.777774 19230 catalog_manager.cc:5697] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 reported cstate change: term changed from 1 to 2, leader changed from c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66) to 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "10584038f61a4dd4a618aa4d75689a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:38.883312 19698 raft_consensus.cc:993] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: : Instructing follower 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:38.883822 19613 raft_consensus.cc:1081] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Signalling peer 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:38.885421 19482 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:34858
I20260709 13:06:38.885960 19482 raft_consensus.cc:493] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:38.886296 19482 raft_consensus.cc:3060] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.891134 19482 raft_consensus.cc:515] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.892728 19482 leader_election.cc:290] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 2 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:38.893492 19335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:38.893806 19409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0ce4fd1f904327b43092aff73b5ce8" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:38.894147 19335 raft_consensus.cc:3060] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.894399 19409 raft_consensus.cc:3055] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:38.894728 19409 raft_consensus.cc:740] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:38.895417 19409 consensus_queue.cc:260] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.896386 19409 raft_consensus.cc:3060] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.899085 19335 raft_consensus.cc:2468] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 2.
I20260709 13:06:38.900285 19449 leader_election.cc:304] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:38.901224 19646 raft_consensus.cc:2804] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:38.901760 19646 raft_consensus.cc:697] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:38.902845 19409 raft_consensus.cc:2468] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 2.
I20260709 13:06:38.902599 19646 consensus_queue.cc:237] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:38.909948 19230 catalog_manager.cc:5697] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 1 to 2, leader changed from c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66) to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:39.039724 19482 raft_consensus.cc:1275] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.041131 19686 consensus_queue.cc:1048] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.049106 19409 raft_consensus.cc:1275] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.050760 19699 consensus_queue.cc:1048] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.086948 19335 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:39.092052 19335 raft_consensus.cc:1275] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.093382 19613 consensus_queue.cc:1048] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.101857 19482 raft_consensus.cc:1275] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer c9592e56d9fa4405a1cdca7ccc6a4e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.103390 19644 consensus_queue.cc:1048] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.315482 19409 raft_consensus.cc:1275] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.316866 19699 consensus_queue.cc:1048] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.326707 19482 raft_consensus.cc:1275] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.327370 19485 raft_consensus.cc:1275] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.329031 19699 consensus_queue.cc:1048] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.329269 19686 consensus_queue.cc:1048] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:39.347220 19408 raft_consensus.cc:1275] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 10584038f61a4dd4a618aa4d75689a05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.349826 19699 consensus_queue.cc:1048] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.396014 19408 raft_consensus.cc:1275] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.397354 19646 consensus_queue.cc:1048] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.408550 19335 raft_consensus.cc:1275] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.409976 19704 consensus_queue.cc:1048] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:41.516748 19233 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58088:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 13:06:41.538873 19233 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58088:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 13:06:41.635133  7481 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 13:06:41.708599 19755 tablet.cc:1724] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:06:41.736172 19300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41545, user_credentials={real_user=slave}} blocked reactor thread for 55950.2us
I20260709 13:06:41.737481 19761 tablet.cc:1724] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.746199 19410 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
mode: GRACEFUL
new_leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:51334
I20260709 13:06:41.782702 19766 tablet.cc:1724] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.796823 19749 tablet.cc:1724] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.686914 19748 tablet.cc:1724] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:06:41.805349 19742 log.cc:927] Time spent T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 13:06:41.807727 19410 raft_consensus.cc:606] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Received request to transfer leadership to TS 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:41.747710 19756 tablet.cc:1724] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.822732 19747 tablet.cc:1724] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.824019 19762 tablet.cc:1724] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.764132 19765 tablet.cc:1724] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:06:41.849526 19511 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:33748250b0014ce08aa43709d62646fc)
I20260709 13:06:41.765897 19745 tablet.cc:1724] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:06:41.868438 19300 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:41545, user_credentials={real_user=slave}} blocked reactor thread for 58778.5us
I20260709 13:06:41.874372 19745 tablet.cc:1724] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:06:41.891110 19375 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.65:37185, user_credentials={real_user=slave}} blocked reactor thread for 83174.5us
W20260709 13:06:41.891688 19511 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:33748250b0014ce08aa43709d62646fc)
W20260709 13:06:41.896086 19511 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:33748250b0014ce08aa43709d62646fc)
I20260709 13:06:41.903687 19755 tablet.cc:1724] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.916373 19747 tablet.cc:1724] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.916358 19745 tablet.cc:1724] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.923043 19761 tablet.cc:1724] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.925357 19748 tablet.cc:1724] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.930788 19768 tablet.cc:1724] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.947153 19774 tablet.cc:1724] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.970956 19749 tablet.cc:1724] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.974254 19334 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
mode: GRACEFUL
new_leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:44390
I20260709 13:06:41.975039 19334 raft_consensus.cc:606] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Received request to transfer leadership to TS 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:41.957105 19756 tablet.cc:1724] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.976719  7481 auto_leader_rebalancer.cc:452] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 13:06:41.977495  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:41.981844 19706 raft_consensus.cc:993] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: : Instructing follower 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:41.982686 19723 raft_consensus.cc:1081] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Signalling peer 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:41.984472 19766 tablet.cc:1724] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:41.998343 19483 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:34858
I20260709 13:06:41.999238 19483 raft_consensus.cc:493] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:42.003921 19483 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.009763 19332 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0bd0066d193143238a07dac629a958b7"
dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
mode: GRACEFUL
new_leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:44390
I20260709 13:06:42.010421 19332 raft_consensus.cc:606] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Received request to transfer leadership to TS 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:42.013888 19406 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566"
dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
mode: GRACEFUL
new_leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:51334
I20260709 13:06:42.014581 19406 raft_consensus.cc:606] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Received request to transfer leadership to TS 2a776fc68e534b9ea0f748905e1de9e8
I20260709 13:06:42.014030 19745 tablet.cc:1724] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:42.016206  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:42.016840  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:42.019305 19747 tablet.cc:1724] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:42.023545 19761 tablet.cc:1724] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:42.009052 19765 tablet.cc:1724] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:42.005911 19762 tablet.cc:1724] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:42.008718 19756 tablet.cc:1724] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:42.052265 19483 raft_consensus.cc:515] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.065997 19332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:42.066654 19332 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.078171 19483 leader_election.cc:290] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 3 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:42.079278 19406 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42afcf5267814a8a97dc8562869ee54f" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:42.080068 19406 raft_consensus.cc:3055] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:42.080417 19406 raft_consensus.cc:740] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:42.081336 19406 consensus_queue.cc:260] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.082722 19406 raft_consensus.cc:3060] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.083519 19332 raft_consensus.cc:2468] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 3.
I20260709 13:06:42.084913 19449 leader_election.cc:304] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:42.085944 19757 raft_consensus.cc:2804] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:42.086436 19757 raft_consensus.cc:697] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:42.087239 19757 consensus_queue.cc:237] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.091374 19406 raft_consensus.cc:2468] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 3.
I20260709 13:06:42.096839 19233 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019: soft_deleted_table [id=fe6283a9f81d4a3f8f20d818e19cc6f7] alter complete (version 1)
I20260709 13:06:42.105377 19231 catalog_manager.cc:5697] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 2 to 3, leader changed from c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66) to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:42.267092 19751 raft_consensus.cc:993] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: : Instructing follower 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:42.267561 19751 raft_consensus.cc:1081] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Signalling peer 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:42.269341 19483 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:34872
I20260709 13:06:42.269889 19483 raft_consensus.cc:493] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:42.270174 19483 raft_consensus.cc:3060] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:42.276885 19483 raft_consensus.cc:515] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.278635 19483 leader_election.cc:290] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 2 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:42.279541 19332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:42.279918 19406 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30a82466431423e9597c9dee074bcf1" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:42.280371 19332 raft_consensus.cc:3055] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:42.280751 19332 raft_consensus.cc:740] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:42.280786 19406 raft_consensus.cc:3060] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:42.282094 19332 consensus_queue.cc:260] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.283421 19332 raft_consensus.cc:3060] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:42.289740 19406 raft_consensus.cc:2468] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 2.
I20260709 13:06:42.291246 19449 leader_election.cc:304] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:42.291356 19332 raft_consensus.cc:2468] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 2.
I20260709 13:06:42.292268 19757 raft_consensus.cc:2804] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:42.292891 19757 raft_consensus.cc:697] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:42.293782 19757 consensus_queue.cc:237] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.302382 19230 catalog_manager.cc:5697] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 1 to 2, leader changed from 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65) to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 2 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:42.462371 19709 raft_consensus.cc:993] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05: : Instructing follower 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:42.462895 19751 raft_consensus.cc:1081] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Signalling peer 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:42.464457 19485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0bd0066d193143238a07dac629a958b7"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:34872
I20260709 13:06:42.464951 19485 raft_consensus.cc:493] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:42.465251 19485 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.470078 19485 raft_consensus.cc:515] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.471791 19485 leader_election.cc:290] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 3 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:42.472529 19332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:42.472873 19406 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd0066d193143238a07dac629a958b7" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:42.473227 19332 raft_consensus.cc:3055] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:42.473507 19406 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.473675 19332 raft_consensus.cc:740] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10584038f61a4dd4a618aa4d75689a05, State: Running, Role: LEADER
I20260709 13:06:42.474480 19332 consensus_queue.cc:260] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.475531 19332 raft_consensus.cc:3060] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.479151 19406 raft_consensus.cc:2468] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 3.
I20260709 13:06:42.480449 19449 leader_election.cc:304] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a776fc68e534b9ea0f748905e1de9e8, c9592e56d9fa4405a1cdca7ccc6a4e28; no voters: 
I20260709 13:06:42.481217 19757 raft_consensus.cc:2804] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:42.481619 19332 raft_consensus.cc:2468] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 3.
I20260709 13:06:42.481947 19757 raft_consensus.cc:697] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:42.482954 19757 consensus_queue.cc:237] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.494784 19230 catalog_manager.cc:5697] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 2 to 3, leader changed from 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65) to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:42.500365 19776 raft_consensus.cc:993] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28: : Instructing follower 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:42.500988 19706 raft_consensus.cc:1081] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Signalling peer 2a776fc68e534b9ea0f748905e1de9e8 to start an election
I20260709 13:06:42.503103 19485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566"
dest_uuid: "2a776fc68e534b9ea0f748905e1de9e8"
 from {username='slave'} at 127.0.0.1:34858
I20260709 13:06:42.504101 19485 raft_consensus.cc:493] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:42.504465 19485 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.509812 19485 raft_consensus.cc:515] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.511409 19485 leader_election.cc:290] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 3 election: Requested vote from peers 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185), c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545)
I20260709 13:06:42.512382 19332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "10584038f61a4dd4a618aa4d75689a05"
I20260709 13:06:42.512569 19406 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a971384ea43f45c0a2753de28bbad566" candidate_uuid: "2a776fc68e534b9ea0f748905e1de9e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28"
I20260709 13:06:42.513106 19332 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.513187 19406 raft_consensus.cc:3055] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:42.513682 19406 raft_consensus.cc:740] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c9592e56d9fa4405a1cdca7ccc6a4e28, State: Running, Role: LEADER
I20260709 13:06:42.514390 19406 consensus_queue.cc:260] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.515427 19406 raft_consensus.cc:3060] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:42.519739 19332 raft_consensus.cc:2468] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 3.
I20260709 13:06:42.521031 19449 leader_election.cc:304] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10584038f61a4dd4a618aa4d75689a05, 2a776fc68e534b9ea0f748905e1de9e8; no voters: 
I20260709 13:06:42.521689 19406 raft_consensus.cc:2468] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a776fc68e534b9ea0f748905e1de9e8 in term 3.
I20260709 13:06:42.521821 19757 raft_consensus.cc:2804] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:42.522416 19757 raft_consensus.cc:697] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 LEADER]: Becoming Leader. State: Replica: 2a776fc68e534b9ea0f748905e1de9e8, State: Running, Role: LEADER
I20260709 13:06:42.523269 19757 consensus_queue.cc:237] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } }
I20260709 13:06:42.531441 19230 catalog_manager.cc:5697] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 reported cstate change: term changed from 2 to 3, leader changed from c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66) to 2a776fc68e534b9ea0f748905e1de9e8 (127.7.78.67). New cstate: current_term: 3 leader_uuid: "2a776fc68e534b9ea0f748905e1de9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a776fc68e534b9ea0f748905e1de9e8" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:42.678887 19332 raft_consensus.cc:1275] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:06:42.680600 19757 consensus_queue.cc:1048] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:42.689101 19406 raft_consensus.cc:1275] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:06:42.691161 19757 consensus_queue.cc:1048] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:42.736352 19406 raft_consensus.cc:1275] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:06:42.737887 19757 consensus_queue.cc:1048] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:42.747332 19332 raft_consensus.cc:1275] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:06:42.750043 19785 consensus_queue.cc:1048] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:42.935727 19334 raft_consensus.cc:1275] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:42.936987 19785 consensus_queue.cc:1048] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:42.947551 19406 raft_consensus.cc:1275] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:42.949229 19757 consensus_queue.cc:1048] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:42.958360 19334 raft_consensus.cc:1275] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:42.960623 19757 consensus_queue.cc:1048] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10584038f61a4dd4a618aa4d75689a05" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:42.975203 19406 raft_consensus.cc:1275] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Refusing update from remote peer 2a776fc68e534b9ea0f748905e1de9e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:42.976624 19646 consensus_queue.cc:1048] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9592e56d9fa4405a1cdca7ccc6a4e28" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:43.081991  7481 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 13:06:43.088393  7481 auto_leader_rebalancer.cc:452] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 13:06:43.088863  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:43.095564  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 13:06:43.096231  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:44.130447  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
W20260709 13:06:44.145473 19449 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
W20260709 13:06:44.161381 19375 consensus_peers.cc:597] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:44.163582 19449 consensus_peers.cc:597] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.173128  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:44.173991  7481 tablet_replica.cc:333] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.174654  7481 raft_consensus.cc:2243] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.175159  7481 raft_consensus.cc:2272] T 1e0ce4fd1f904327b43092aff73b5ce8 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.177459  7481 tablet_replica.cc:333] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.178231  7481 raft_consensus.cc:2243] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.178845  7481 raft_consensus.cc:2272] T 33748250b0014ce08aa43709d62646fc P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.181128  7481 tablet_replica.cc:333] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.181772  7481 raft_consensus.cc:2243] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.182252  7481 raft_consensus.cc:2272] T 1a5b17c161694271849e71bb7476cda3 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.184168  7481 tablet_replica.cc:333] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.184770  7481 raft_consensus.cc:2243] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.185192  7481 raft_consensus.cc:2272] T a30a82466431423e9597c9dee074bcf1 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.187155  7481 tablet_replica.cc:333] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.187914  7481 raft_consensus.cc:2243] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.188788  7481 raft_consensus.cc:2272] T 5b68875e4cde4edf9c9840b172043454 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.190744  7481 tablet_replica.cc:333] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.191306  7481 raft_consensus.cc:2243] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.192077  7481 raft_consensus.cc:2272] T 95cb88079f21459cac0767e8a11ed708 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.194007  7481 tablet_replica.cc:333] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.194566  7481 raft_consensus.cc:2243] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.195019  7481 raft_consensus.cc:2272] T 42afcf5267814a8a97dc8562869ee54f P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.197031  7481 tablet_replica.cc:333] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.197757  7481 raft_consensus.cc:2243] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.198632  7481 raft_consensus.cc:2272] T fc4f89a87a274604883f9b4f91d9872b P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.200681  7481 tablet_replica.cc:333] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.201261  7481 raft_consensus.cc:2243] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.202219  7481 raft_consensus.cc:2272] T 468cc51b6ba9441d8a566322b6e6bcbc P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.204432  7481 tablet_replica.cc:333] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.205035  7481 raft_consensus.cc:2243] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.205559  7481 raft_consensus.cc:2272] T e05e4b201969463ea0a99d83676e1b23 P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.207816  7481 tablet_replica.cc:333] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.208388  7481 raft_consensus.cc:2243] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.208863  7481 raft_consensus.cc:2272] T a971384ea43f45c0a2753de28bbad566 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.211338  7481 tablet_replica.cc:333] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.211978  7481 raft_consensus.cc:2243] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.212731  7481 raft_consensus.cc:2272] T 4e07978d5df643538c4d45a6a7ae4f47 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.215433  7481 tablet_replica.cc:333] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.216130  7481 raft_consensus.cc:2243] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.216609  7481 raft_consensus.cc:2272] T eb10d732a4834e1cbd9d4eb315bcb0dd P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.219003  7481 tablet_replica.cc:333] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.219527  7481 raft_consensus.cc:2243] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.219961  7481 raft_consensus.cc:2272] T 2efe92bdb3104343a4545e1b015c9fce P 10584038f61a4dd4a618aa4d75689a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.221897  7481 tablet_replica.cc:333] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.222453  7481 raft_consensus.cc:2243] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.222879  7481 raft_consensus.cc:2272] T 0bd0066d193143238a07dac629a958b7 P 10584038f61a4dd4a618aa4d75689a05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.224834  7481 tablet_replica.cc:333] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.225560  7481 raft_consensus.cc:2243] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.226016  7481 raft_consensus.cc:2272] T 26edbcff77f1411481183330d339edd8 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.228042  7481 tablet_replica.cc:333] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.228533  7481 raft_consensus.cc:2243] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.229296  7481 raft_consensus.cc:2272] T b843b9716ea0488b94a41b935f80f794 P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.231285  7481 tablet_replica.cc:333] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05: stopping tablet replica
I20260709 13:06:44.231865  7481 raft_consensus.cc:2243] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.232342  7481 raft_consensus.cc:2272] T b8afd9e635b348b0938920cd55a0615a P 10584038f61a4dd4a618aa4d75689a05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.274071  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
I20260709 13:06:44.307245  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
W20260709 13:06:44.332472 19449 consensus_peers.cc:597] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Couldn't send request to peer c9592e56d9fa4405a1cdca7ccc6a4e28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.339804  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:44.340724  7481 tablet_replica.cc:333] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.341557  7481 raft_consensus.cc:2243] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.342578  7481 raft_consensus.cc:2272] T 33748250b0014ce08aa43709d62646fc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.344973  7481 tablet_replica.cc:333] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.345659  7481 raft_consensus.cc:2243] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:44.345669 19375 consensus_peers.cc:597] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.346325  7481 raft_consensus.cc:2272] T 42afcf5267814a8a97dc8562869ee54f P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.348328  7481 tablet_replica.cc:333] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.348887  7481 raft_consensus.cc:2243] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.349301  7481 raft_consensus.cc:2272] T a30a82466431423e9597c9dee074bcf1 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.351202  7481 tablet_replica.cc:333] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.351742  7481 raft_consensus.cc:2243] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.352142  7481 raft_consensus.cc:2272] T 1e0ce4fd1f904327b43092aff73b5ce8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.354066  7481 tablet_replica.cc:333] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.354619  7481 raft_consensus.cc:2243] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.355031  7481 raft_consensus.cc:2272] T 5b68875e4cde4edf9c9840b172043454 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.356879  7481 tablet_replica.cc:333] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.357570  7481 raft_consensus.cc:2243] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.358863  7481 raft_consensus.cc:2272] T 1a5b17c161694271849e71bb7476cda3 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.361317  7481 tablet_replica.cc:333] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.362008  7481 raft_consensus.cc:2243] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.362460  7481 raft_consensus.cc:2272] T 95cb88079f21459cac0767e8a11ed708 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.364464  7481 tablet_replica.cc:333] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.365089  7481 raft_consensus.cc:2243] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.365623  7481 raft_consensus.cc:2272] T b843b9716ea0488b94a41b935f80f794 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.368140  7481 tablet_replica.cc:333] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.368849  7481 raft_consensus.cc:2243] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.369485  7481 raft_consensus.cc:2272] T b8afd9e635b348b0938920cd55a0615a P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:44.370702 19449 consensus_peers.cc:597] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Couldn't send request to peer c9592e56d9fa4405a1cdca7ccc6a4e28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.371933  7481 tablet_replica.cc:333] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.372622  7481 raft_consensus.cc:2243] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.373721  7481 raft_consensus.cc:2272] T 26edbcff77f1411481183330d339edd8 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.376209  7481 tablet_replica.cc:333] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.376878  7481 raft_consensus.cc:2243] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.377449  7481 raft_consensus.cc:2272] T 468cc51b6ba9441d8a566322b6e6bcbc P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:44.378170 19449 consensus_peers.cc:597] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.379921  7481 tablet_replica.cc:333] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.380585  7481 raft_consensus.cc:2243] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.381047  7481 raft_consensus.cc:2272] T fc4f89a87a274604883f9b4f91d9872b P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.383505  7481 tablet_replica.cc:333] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.384258  7481 raft_consensus.cc:2243] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.384831  7481 raft_consensus.cc:2272] T 4e07978d5df643538c4d45a6a7ae4f47 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.387531  7481 tablet_replica.cc:333] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.388294  7481 raft_consensus.cc:2243] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 LEADER]: Raft consensus shutting down.
W20260709 13:06:44.389128 19375 consensus_peers.cc:597] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.389338  7481 raft_consensus.cc:2272] T eb10d732a4834e1cbd9d4eb315bcb0dd P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.391613  7481 tablet_replica.cc:333] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.392194  7481 raft_consensus.cc:2243] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.392661  7481 raft_consensus.cc:2272] T a971384ea43f45c0a2753de28bbad566 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.394670  7481 tablet_replica.cc:333] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.395222  7481 raft_consensus.cc:2243] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.396010  7481 raft_consensus.cc:2272] T e05e4b201969463ea0a99d83676e1b23 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.398099  7481 tablet_replica.cc:333] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.398602  7481 raft_consensus.cc:2243] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.399356  7481 raft_consensus.cc:2272] T 2efe92bdb3104343a4545e1b015c9fce P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.401250  7481 tablet_replica.cc:333] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28: stopping tablet replica
I20260709 13:06:44.401813  7481 raft_consensus.cc:2243] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.402231  7481 raft_consensus.cc:2272] T 0bd0066d193143238a07dac629a958b7 P c9592e56d9fa4405a1cdca7ccc6a4e28 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:44.418061 19449 consensus_peers.cc:597] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:44.421403 19449 consensus_peers.cc:597] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Couldn't send request to peer c9592e56d9fa4405a1cdca7ccc6a4e28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.443825  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
I20260709 13:06:44.478909  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
W20260709 13:06:44.487392 19449 consensus_peers.cc:597] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:44.491709 19449 consensus_peers.cc:597] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer 10584038f61a4dd4a618aa4d75689a05 (127.7.78.65:37185): Couldn't send request to peer 10584038f61a4dd4a618aa4d75689a05. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.510768  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:44.511618  7481 tablet_replica.cc:333] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.512492  7481 raft_consensus.cc:2243] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.513037  7481 raft_consensus.cc:2272] T 33748250b0014ce08aa43709d62646fc P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.515369  7481 tablet_replica.cc:333] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.515990  7481 raft_consensus.cc:2243] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.516491  7481 raft_consensus.cc:2272] T 1a5b17c161694271849e71bb7476cda3 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.519346  7481 tablet_replica.cc:333] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.520095  7481 raft_consensus.cc:2243] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.521183  7481 raft_consensus.cc:2272] T 42afcf5267814a8a97dc8562869ee54f P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:44.521865 19449 consensus_peers.cc:597] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 -> Peer c9592e56d9fa4405a1cdca7ccc6a4e28 (127.7.78.66:41545): Couldn't send request to peer c9592e56d9fa4405a1cdca7ccc6a4e28. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:41545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:44.523530  7481 tablet_replica.cc:333] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.524225  7481 raft_consensus.cc:2243] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.525013  7481 raft_consensus.cc:2272] T a30a82466431423e9597c9dee074bcf1 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.527226  7481 tablet_replica.cc:333] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.527827  7481 raft_consensus.cc:2243] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.528304  7481 raft_consensus.cc:2272] T 95cb88079f21459cac0767e8a11ed708 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.530306  7481 tablet_replica.cc:333] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.530844  7481 raft_consensus.cc:2243] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.531628  7481 raft_consensus.cc:2272] T 1e0ce4fd1f904327b43092aff73b5ce8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.533639  7481 tablet_replica.cc:333] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.534209  7481 raft_consensus.cc:2243] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.534688  7481 raft_consensus.cc:2272] T 5b68875e4cde4edf9c9840b172043454 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.536944  7481 tablet_replica.cc:333] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.537667  7481 raft_consensus.cc:2243] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.538199  7481 raft_consensus.cc:2272] T fc4f89a87a274604883f9b4f91d9872b P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.540732  7481 tablet_replica.cc:333] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.541333  7481 raft_consensus.cc:2243] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.541930  7481 raft_consensus.cc:2272] T 468cc51b6ba9441d8a566322b6e6bcbc P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.544085  7481 tablet_replica.cc:333] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.544770  7481 raft_consensus.cc:2243] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.545258  7481 raft_consensus.cc:2272] T b843b9716ea0488b94a41b935f80f794 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.547430  7481 tablet_replica.cc:333] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.548039  7481 raft_consensus.cc:2243] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.548895  7481 raft_consensus.cc:2272] T b8afd9e635b348b0938920cd55a0615a P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.551101  7481 tablet_replica.cc:333] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.551692  7481 raft_consensus.cc:2243] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.552163  7481 raft_consensus.cc:2272] T 26edbcff77f1411481183330d339edd8 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.554457  7481 tablet_replica.cc:333] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.555051  7481 raft_consensus.cc:2243] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.555523  7481 raft_consensus.cc:2272] T 4e07978d5df643538c4d45a6a7ae4f47 P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.557763  7481 tablet_replica.cc:333] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.558362  7481 raft_consensus.cc:2243] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.558867  7481 raft_consensus.cc:2272] T eb10d732a4834e1cbd9d4eb315bcb0dd P 2a776fc68e534b9ea0f748905e1de9e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.561061  7481 tablet_replica.cc:333] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.561673  7481 raft_consensus.cc:2243] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.562526  7481 raft_consensus.cc:2272] T a971384ea43f45c0a2753de28bbad566 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.565025  7481 tablet_replica.cc:333] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.565652  7481 raft_consensus.cc:2243] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.566130  7481 raft_consensus.cc:2272] T 2efe92bdb3104343a4545e1b015c9fce P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.568269  7481 tablet_replica.cc:333] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.568845  7481 raft_consensus.cc:2243] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.569653  7481 raft_consensus.cc:2272] T 0bd0066d193143238a07dac629a958b7 P 2a776fc68e534b9ea0f748905e1de9e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.571795  7481 tablet_replica.cc:333] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8: stopping tablet replica
I20260709 13:06:44.572337  7481 raft_consensus.cc:2243] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:44.572791  7481 raft_consensus.cc:2272] T e05e4b201969463ea0a99d83676e1b23 P 2a776fc68e534b9ea0f748905e1de9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.613581  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
I20260709 13:06:44.647775  7481 master.cc:562] Master@127.7.78.126:40605 shutting down...
I20260709 13:06:44.673672  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:44.674223  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:44.674528  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3b5c046b9f3344a08ae2144554581019: stopping tablet replica
I20260709 13:06:44.695386  7481 master.cc:584] Master@127.7.78.126:40605 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (12940 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 13:06:44.749210  7481 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.78.126:33377
I20260709 13:06:44.750337  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:44.756129 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:44.756247 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:44.757234 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:44.757656  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:44.758657  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:44.758872  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:44.759057  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602404759037 us; error 0 us; skew 500 ppm
I20260709 13:06:44.759588  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:44.762109  7481 webserver.cc:533] Webserver started at http://127.7.78.126:38459/ using document root <none> and password file <none>
I20260709 13:06:44.762598  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:44.762802  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:44.763075  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:44.764173  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/master-0-root/instance:
uuid: "edada50e89b24f0781f15a17e0d28107"
format_stamp: "Formatted at 2026-07-09 13:06:44 on dist-test-slave-50cs"
I20260709 13:06:44.768850  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260709 13:06:44.772394 19804 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:44.773342  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 13:06:44.773700  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/master-0-root
uuid: "edada50e89b24f0781f15a17e0d28107"
format_stamp: "Formatted at 2026-07-09 13:06:44 on dist-test-slave-50cs"
I20260709 13:06:44.773959  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:44.793254  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:44.795158  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:44.842145  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.126:33377
I20260709 13:06:44.842406 19855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.126:33377 every 8 connection(s)
I20260709 13:06:44.846282 19856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:44.857640 19856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107: Bootstrap starting.
I20260709 13:06:44.862670 19856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:44.867151 19856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107: No bootstrap required, opened a new log
I20260709 13:06:44.869470 19856 raft_consensus.cc:359] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edada50e89b24f0781f15a17e0d28107" member_type: VOTER }
I20260709 13:06:44.869967 19856 raft_consensus.cc:385] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:44.870280 19856 raft_consensus.cc:740] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edada50e89b24f0781f15a17e0d28107, State: Initialized, Role: FOLLOWER
I20260709 13:06:44.871021 19856 consensus_queue.cc:260] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edada50e89b24f0781f15a17e0d28107" member_type: VOTER }
I20260709 13:06:44.871507 19856 raft_consensus.cc:399] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:44.871747 19856 raft_consensus.cc:493] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:44.872058 19856 raft_consensus.cc:3060] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:44.877218 19856 raft_consensus.cc:515] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edada50e89b24f0781f15a17e0d28107" member_type: VOTER }
I20260709 13:06:44.877823 19856 leader_election.cc:304] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edada50e89b24f0781f15a17e0d28107; no voters: 
I20260709 13:06:44.878962 19856 leader_election.cc:290] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:44.879318 19859 raft_consensus.cc:2804] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:44.880800 19859 raft_consensus.cc:697] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 1 LEADER]: Becoming Leader. State: Replica: edada50e89b24f0781f15a17e0d28107, State: Running, Role: LEADER
I20260709 13:06:44.881533 19859 consensus_queue.cc:237] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edada50e89b24f0781f15a17e0d28107" member_type: VOTER }
I20260709 13:06:44.882030 19856 sys_catalog.cc:565] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:44.886250 19860 sys_catalog.cc:455] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edada50e89b24f0781f15a17e0d28107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edada50e89b24f0781f15a17e0d28107" member_type: VOTER } }
I20260709 13:06:44.886332 19861 sys_catalog.cc:455] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edada50e89b24f0781f15a17e0d28107. Latest consensus state: current_term: 1 leader_uuid: "edada50e89b24f0781f15a17e0d28107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edada50e89b24f0781f15a17e0d28107" member_type: VOTER } }
I20260709 13:06:44.886960 19860 sys_catalog.cc:458] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:44.887198 19861 sys_catalog.cc:458] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:44.890240 19866 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:44.895157 19866 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:44.901798  7481 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:44.905591 19866 catalog_manager.cc:1383] Generated new cluster ID: 88b9941e5ab74cc49e837d0d2479ddab
I20260709 13:06:44.905912 19866 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:44.933074 19866 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:44.934690 19866 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:44.949985 19866 catalog_manager.cc:6070] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107: Generated new TSK 0
I20260709 13:06:44.950637 19866 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:44.972095  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:44.978826 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:44.979763 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:44.984202 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:44.986124  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:44.987995  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:44.988188  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:44.988394  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602404988379 us; error 0 us; skew 500 ppm
I20260709 13:06:44.988974  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:44.991350  7481 webserver.cc:533] Webserver started at http://127.7.78.65:44789/ using document root <none> and password file <none>
I20260709 13:06:44.991842  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:44.992039  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:44.992323  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:44.993449  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-0-root/instance:
uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
format_stamp: "Formatted at 2026-07-09 13:06:44 on dist-test-slave-50cs"
I20260709 13:06:44.998107  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 13:06:45.001909 19885 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.002847  7481 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 13:06:45.003118  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-0-root
uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
format_stamp: "Formatted at 2026-07-09 13:06:44 on dist-test-slave-50cs"
I20260709 13:06:45.003378  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:45.028455  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:45.029709  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:45.031159  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:45.033619  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:45.033810  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.034123  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:45.034303  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.076452  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.65:44645
I20260709 13:06:45.076532 19947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.65:44645 every 8 connection(s)
I20260709 13:06:45.081782  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:45.088999 19952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:45.090736 19953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:45.093997  7481 server_base.cc:1061] running on GCE node
W20260709 13:06:45.094563 19955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:45.095806  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:45.096028  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:45.096158  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602405096147 us; error 0 us; skew 500 ppm
I20260709 13:06:45.096680  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:45.100626  7481 webserver.cc:533] Webserver started at http://127.7.78.66:42823/ using document root <none> and password file <none>
I20260709 13:06:45.101238  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:45.101476  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:45.101800  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:45.101861 19948 heartbeater.cc:344] Connected to a master server at 127.7.78.126:33377
I20260709 13:06:45.102360 19948 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:45.102986  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-1-root/instance:
uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
format_stamp: "Formatted at 2026-07-09 13:06:45 on dist-test-slave-50cs"
I20260709 13:06:45.103082 19948 heartbeater.cc:507] Master 127.7.78.126:33377 requested a full tablet report, sending...
I20260709 13:06:45.105535 19821 ts_manager.cc:194] Registered new tserver with Master: 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645)
I20260709 13:06:45.107506 19821 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50446
I20260709 13:06:45.108166  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260709 13:06:45.111889 19960 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.112948  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260709 13:06:45.113282  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-1-root
uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
format_stamp: "Formatted at 2026-07-09 13:06:45 on dist-test-slave-50cs"
I20260709 13:06:45.113631  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:45.132090  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:45.133234  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:45.134748  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:45.137049  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:45.137286  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.137553  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:45.137732  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.178136  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.66:35681
I20260709 13:06:45.178218 20022 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.66:35681 every 8 connection(s)
I20260709 13:06:45.183311  7481 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:45.190873 20026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:45.192078 20027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:45.194672 20029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:45.195421  7481 server_base.cc:1061] running on GCE node
I20260709 13:06:45.196766  7481 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:45.197065  7481 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:45.197326  7481 hybrid_clock.cc:648] HybridClock initialized: now 1783602405197305 us; error 0 us; skew 500 ppm
I20260709 13:06:45.198127  7481 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:45.198483 20023 heartbeater.cc:344] Connected to a master server at 127.7.78.126:33377
I20260709 13:06:45.199079 20023 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:45.200083 20023 heartbeater.cc:507] Master 127.7.78.126:33377 requested a full tablet report, sending...
I20260709 13:06:45.201670  7481 webserver.cc:533] Webserver started at http://127.7.78.67:34199/ using document root <none> and password file <none>
I20260709 13:06:45.202417 19821 ts_manager.cc:194] Registered new tserver with Master: 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:45.202450  7481 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:45.202939  7481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:45.203286  7481 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:45.203946 19821 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50458
I20260709 13:06:45.204632  7481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-2-root/instance:
uuid: "d74830b4b12e4c6f938dac068fff593d"
format_stamp: "Formatted at 2026-07-09 13:06:45 on dist-test-slave-50cs"
I20260709 13:06:45.210531  7481 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20260709 13:06:45.214022 20034 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.214771  7481 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 13:06:45.215083  7481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-2-root
uuid: "d74830b4b12e4c6f938dac068fff593d"
format_stamp: "Formatted at 2026-07-09 13:06:45 on dist-test-slave-50cs"
I20260709 13:06:45.215431  7481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskPHGGl2/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602031409637-7481-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:45.231492  7481 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:45.232738  7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:45.234495  7481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:45.236755  7481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:45.237004  7481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.237250  7481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:45.237448  7481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:45.282330  7481 rpc_server.cc:307] RPC server started. Bound to: 127.7.78.67:36893
I20260709 13:06:45.282536 20096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.78.67:36893 every 8 connection(s)
I20260709 13:06:45.297240 20097 heartbeater.cc:344] Connected to a master server at 127.7.78.126:33377
I20260709 13:06:45.297856 20097 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:45.298642 20097 heartbeater.cc:507] Master 127.7.78.126:33377 requested a full tablet report, sending...
I20260709 13:06:45.300680 19821 ts_manager.cc:194] Registered new tserver with Master: d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:45.301596  7481 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015767923s
I20260709 13:06:45.302167 19821 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50472
I20260709 13:06:45.328570 19821 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:45.331274 19821 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:45.408609 20062 tablet_service.cc:1511] Processing CreateTablet for tablet b051a1e1b9f04842802d2af08b3bbe9b (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:45.410162 20062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b051a1e1b9f04842802d2af08b3bbe9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.419198 20061 tablet_service.cc:1511] Processing CreateTablet for tablet 4695d39566674c9a9dadb0c3ef9925f1 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:45.419920 19913 tablet_service.cc:1511] Processing CreateTablet for tablet b051a1e1b9f04842802d2af08b3bbe9b (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:45.420828 20061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4695d39566674c9a9dadb0c3ef9925f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.421430 19913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b051a1e1b9f04842802d2af08b3bbe9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.425472 19911 tablet_service.cc:1511] Processing CreateTablet for tablet a2c7846e41f249af96a83b97e089f89f (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:45.426967 19911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c7846e41f249af96a83b97e089f89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.427492 20060 tablet_service.cc:1511] Processing CreateTablet for tablet a2c7846e41f249af96a83b97e089f89f (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:45.428465 19912 tablet_service.cc:1511] Processing CreateTablet for tablet 4695d39566674c9a9dadb0c3ef9925f1 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:45.428977 20060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c7846e41f249af96a83b97e089f89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.430656 19910 tablet_service.cc:1511] Processing CreateTablet for tablet 51d91599e9b540e3a3c7a75ea0d5a338 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:45.432833 19910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d91599e9b540e3a3c7a75ea0d5a338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.434149 19912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4695d39566674c9a9dadb0c3ef9925f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.438553 19909 tablet_service.cc:1511] Processing CreateTablet for tablet 5c1e2720a76543aaad2d7e42e2c14336 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:45.438793 20059 tablet_service.cc:1511] Processing CreateTablet for tablet 51d91599e9b540e3a3c7a75ea0d5a338 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:45.439218 20058 tablet_service.cc:1511] Processing CreateTablet for tablet 5c1e2720a76543aaad2d7e42e2c14336 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:45.440142 19909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1e2720a76543aaad2d7e42e2c14336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.440271 20059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d91599e9b540e3a3c7a75ea0d5a338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.441057 20058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1e2720a76543aaad2d7e42e2c14336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.442566 19908 tablet_service.cc:1511] Processing CreateTablet for tablet 90cbe74d298c4e6c848fab0a23ed0917 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:45.445667 20057 tablet_service.cc:1511] Processing CreateTablet for tablet 90cbe74d298c4e6c848fab0a23ed0917 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:45.445889 19907 tablet_service.cc:1511] Processing CreateTablet for tablet 87c113c7eeba4c09ab636538d2add58a (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:45.447341 20057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90cbe74d298c4e6c848fab0a23ed0917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.447517 19907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c113c7eeba4c09ab636538d2add58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.449769 19908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90cbe74d298c4e6c848fab0a23ed0917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.452229 19906 tablet_service.cc:1511] Processing CreateTablet for tablet 32b285df036b4239b948bab122c4b8a5 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:45.453852 19906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32b285df036b4239b948bab122c4b8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.452451 20056 tablet_service.cc:1511] Processing CreateTablet for tablet 87c113c7eeba4c09ab636538d2add58a (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:45.458967 20056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c113c7eeba4c09ab636538d2add58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.465651 20055 tablet_service.cc:1511] Processing CreateTablet for tablet 32b285df036b4239b948bab122c4b8a5 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:45.466152 19905 tablet_service.cc:1511] Processing CreateTablet for tablet 18f26dbbb33b4bc096697663cb163f69 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:45.467247 20055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32b285df036b4239b948bab122c4b8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.467777 19905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f26dbbb33b4bc096697663cb163f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.479209 20054 tablet_service.cc:1511] Processing CreateTablet for tablet 18f26dbbb33b4bc096697663cb163f69 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:45.480780 20054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f26dbbb33b4bc096697663cb163f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.498575 19987 tablet_service.cc:1511] Processing CreateTablet for tablet 4695d39566674c9a9dadb0c3ef9925f1 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:45.500242 19987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4695d39566674c9a9dadb0c3ef9925f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.502760 19984 tablet_service.cc:1511] Processing CreateTablet for tablet 5c1e2720a76543aaad2d7e42e2c14336 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:45.504318 19984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1e2720a76543aaad2d7e42e2c14336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.506143 19985 tablet_service.cc:1511] Processing CreateTablet for tablet 51d91599e9b540e3a3c7a75ea0d5a338 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:45.507696 19985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d91599e9b540e3a3c7a75ea0d5a338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.510598 19986 tablet_service.cc:1511] Processing CreateTablet for tablet a2c7846e41f249af96a83b97e089f89f (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:45.512077 19986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c7846e41f249af96a83b97e089f89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.515461 19988 tablet_service.cc:1511] Processing CreateTablet for tablet b051a1e1b9f04842802d2af08b3bbe9b (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:45.516969 19988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b051a1e1b9f04842802d2af08b3bbe9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.502835 19982 tablet_service.cc:1511] Processing CreateTablet for tablet 87c113c7eeba4c09ab636538d2add58a (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:45.500274 19983 tablet_service.cc:1511] Processing CreateTablet for tablet 90cbe74d298c4e6c848fab0a23ed0917 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:45.520787 19983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90cbe74d298c4e6c848fab0a23ed0917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.524398 19982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c113c7eeba4c09ab636538d2add58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.570508 20114 tablet_bootstrap.cc:492] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:45.592334 20113 tablet_bootstrap.cc:492] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:45.594584 20114 tablet_bootstrap.cc:654] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.605098 20113 tablet_bootstrap.cc:654] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.608850 20115 tablet_bootstrap.cc:492] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:45.610668 20114 tablet_bootstrap.cc:492] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:45.651829 19987 tablet_service.cc:1511] Processing CreateTablet for tablet 32b285df036b4239b948bab122c4b8a5 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:45.666225 19987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32b285df036b4239b948bab122c4b8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.662081 19988 tablet_service.cc:1511] Processing CreateTablet for tablet 18f26dbbb33b4bc096697663cb163f69 (DEFAULT_TABLE table=test-workload [id=8c72d56c88c64ab8945fa5c9b129cb69]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:45.679653 19988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f26dbbb33b4bc096697663cb163f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:45.688573 20115 tablet_bootstrap.cc:654] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.689694 20113 tablet_bootstrap.cc:492] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:45.611305 20114 ts_tablet_manager.cc:1403] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.041s	user 0.014s	sys 0.000s
I20260709 13:06:45.706454 20113 ts_tablet_manager.cc:1403] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.114s	user 0.010s	sys 0.009s
I20260709 13:06:45.716888 20113 raft_consensus.cc:359] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.718535 20113 raft_consensus.cc:385] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.719035 20113 raft_consensus.cc:740] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.712589 20114 raft_consensus.cc:359] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.720657 20114 raft_consensus.cc:385] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.725011 20114 raft_consensus.cc:740] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.726436 20114 consensus_queue.cc:260] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.728349 20115 tablet_bootstrap.cc:492] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:45.729187 19948 heartbeater.cc:499] Master 127.7.78.126:33377 was elected leader, sending a full tablet report...
I20260709 13:06:45.729264 20115 ts_tablet_manager.cc:1403] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.121s	user 0.013s	sys 0.010s
I20260709 13:06:45.730017 20113 consensus_queue.cc:260] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.733165 20114 ts_tablet_manager.cc:1434] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.004s
I20260709 13:06:45.733693 20115 raft_consensus.cc:359] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.734614 20115 raft_consensus.cc:385] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.734625 20114 tablet_bootstrap.cc:492] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:45.735054 20115 raft_consensus.cc:740] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.736265 20115 consensus_queue.cc:260] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.746076 20115 ts_tablet_manager.cc:1434] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.016s	user 0.011s	sys 0.003s
I20260709 13:06:45.746717 20113 ts_tablet_manager.cc:1434] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.005s
I20260709 13:06:45.747188 20023 heartbeater.cc:499] Master 127.7.78.126:33377 was elected leader, sending a full tablet report...
I20260709 13:06:45.748088 20097 heartbeater.cc:499] Master 127.7.78.126:33377 was elected leader, sending a full tablet report...
I20260709 13:06:45.757654 20113 tablet_bootstrap.cc:492] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:45.765136 20115 tablet_bootstrap.cc:492] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:45.768777 20114 tablet_bootstrap.cc:654] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.786326 20113 tablet_bootstrap.cc:654] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.787775 20115 tablet_bootstrap.cc:654] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.792583 20114 tablet_bootstrap.cc:492] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:45.793324 20114 ts_tablet_manager.cc:1403] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.059s	user 0.012s	sys 0.001s
I20260709 13:06:45.796242 20115 tablet_bootstrap.cc:492] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:45.796240 20113 tablet_bootstrap.cc:492] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:45.797087 20115 ts_tablet_manager.cc:1403] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.005s
I20260709 13:06:45.797151 20113 ts_tablet_manager.cc:1403] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.040s	user 0.012s	sys 0.004s
I20260709 13:06:45.796861 20114 raft_consensus.cc:359] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:45.797973 20114 raft_consensus.cc:385] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.798321 20114 raft_consensus.cc:740] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.799199 20114 consensus_queue.cc:260] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:45.800365 20115 raft_consensus.cc:359] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:45.801177 20115 raft_consensus.cc:385] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.800879 20113 raft_consensus.cc:359] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.801532 20115 raft_consensus.cc:740] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.801570 20114 ts_tablet_manager.cc:1434] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:45.801774 20113 raft_consensus.cc:385] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.802318 20113 raft_consensus.cc:740] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.803117 20114 tablet_bootstrap.cc:492] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:45.802691 20115 consensus_queue.cc:260] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:45.803589 20113 consensus_queue.cc:260] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.811653 20115 ts_tablet_manager.cc:1434] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 13:06:45.811694 20114 tablet_bootstrap.cc:654] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.812922 20115 tablet_bootstrap.cc:492] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:45.823117 20113 ts_tablet_manager.cc:1434] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.003s
I20260709 13:06:45.823223 20115 tablet_bootstrap.cc:654] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.826227 20113 tablet_bootstrap.cc:492] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:45.837119 20115 tablet_bootstrap.cc:492] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:45.837574 20113 tablet_bootstrap.cc:654] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.837759 20115 ts_tablet_manager.cc:1403] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.007s
I20260709 13:06:45.840492 20114 tablet_bootstrap.cc:492] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:45.841056 20114 ts_tablet_manager.cc:1403] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.038s	user 0.024s	sys 0.012s
I20260709 13:06:45.841603 20115 raft_consensus.cc:359] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.842399 20115 raft_consensus.cc:385] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.842694 20115 raft_consensus.cc:740] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.843856 20113 tablet_bootstrap.cc:492] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:45.843629 20115 consensus_queue.cc:260] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.843992 20114 raft_consensus.cc:359] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.844446 20113 ts_tablet_manager.cc:1403] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.018s	user 0.009s	sys 0.005s
I20260709 13:06:45.844790 20114 raft_consensus.cc:385] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.845110 20114 raft_consensus.cc:740] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.846079 20114 consensus_queue.cc:260] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.847082 20115 ts_tablet_manager.cc:1434] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.001s
I20260709 13:06:45.848318 20115 tablet_bootstrap.cc:492] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:45.847945 20113 raft_consensus.cc:359] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.848826 20113 raft_consensus.cc:385] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.849164 20113 raft_consensus.cc:740] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.850136 20113 consensus_queue.cc:260] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.851716 20114 ts_tablet_manager.cc:1434] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 13:06:45.852988 20114 tablet_bootstrap.cc:492] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:45.856454 20113 ts_tablet_manager.cc:1434] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:06:45.856868 20115 tablet_bootstrap.cc:654] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.857784 20113 tablet_bootstrap.cc:492] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:45.864739 20120 raft_consensus.cc:493] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:45.865444 20120 raft_consensus.cc:515] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.868232 20114 tablet_bootstrap.cc:654] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.873842 20115 tablet_bootstrap.cc:492] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:45.874547 20115 ts_tablet_manager.cc:1403] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.012s
I20260709 13:06:45.877081 20114 tablet_bootstrap.cc:492] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:45.877717 20114 ts_tablet_manager.cc:1403] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.000s
I20260709 13:06:45.878321 20115 raft_consensus.cc:359] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.879189 20115 raft_consensus.cc:385] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.879515 20115 raft_consensus.cc:740] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.880364 20115 consensus_queue.cc:260] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.881230 20114 raft_consensus.cc:359] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.882058 20114 raft_consensus.cc:385] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.882354 20114 raft_consensus.cc:740] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.882908 20115 ts_tablet_manager.cc:1434] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.004s
I20260709 13:06:45.883342 20114 consensus_queue.cc:260] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.884436 20115 tablet_bootstrap.cc:492] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:45.884392 20120 leader_election.cc:290] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:45.894104 20114 ts_tablet_manager.cc:1434] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 13:06:45.894284 20115 tablet_bootstrap.cc:654] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:45.896225 19869 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:45.898424 20113 tablet_bootstrap.cc:654] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.904390 20121 raft_consensus.cc:493] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:45.904683 20119 raft_consensus.cc:493] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:45.905192 20121 raft_consensus.cc:515] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.905551 20119 raft_consensus.cc:515] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.910429 20121 leader_election.cc:290] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:45.912041 20114 tablet_bootstrap.cc:492] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:45.924998 20119 leader_election.cc:290] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:45.945387 20113 tablet_bootstrap.cc:492] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:45.949437 20113 ts_tablet_manager.cc:1403] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.092s	user 0.012s	sys 0.002s
I20260709 13:06:45.954581 20113 raft_consensus.cc:359] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.955816 20113 raft_consensus.cc:385] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.956252 20113 raft_consensus.cc:740] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.974442 20115 tablet_bootstrap.cc:492] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:45.975152 20115 ts_tablet_manager.cc:1403] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.091s	user 0.032s	sys 0.025s
I20260709 13:06:45.957348 20113 consensus_queue.cc:260] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.978888 20115 raft_consensus.cc:359] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.980185 20115 raft_consensus.cc:385] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:45.980543 20115 raft_consensus.cc:740] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:45.981325 20115 consensus_queue.cc:260] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:45.982141 20114 tablet_bootstrap.cc:654] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:45.983783 20115 ts_tablet_manager.cc:1434] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.006s
I20260709 13:06:45.985100 20115 tablet_bootstrap.cc:492] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:45.983759 20113 ts_tablet_manager.cc:1434] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.005s
I20260709 13:06:45.988547 20113 tablet_bootstrap.cc:492] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:46.000365 20120 raft_consensus.cc:493] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.001468 20120 raft_consensus.cc:515] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.001518 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
W20260709 13:06:46.003449 19963 leader_election.cc:343] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.005744 19998 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.006181 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.005110 20120 leader_election.cc:290] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.006605 19998 raft_consensus.cc:2468] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:46.006960 19997 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:46.008869 20036 leader_election.cc:304] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.009970 20036 leader_election.cc:304] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.010489 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.010524 20115 tablet_bootstrap.cc:654] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.011492 19922 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:46.014460 20120 raft_consensus.cc:2804] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.015533 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.015240 20120 raft_consensus.cc:493] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.016230 19923 raft_consensus.cc:2468] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:46.016352 20120 raft_consensus.cc:3060] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.019930 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.021318 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
W20260709 13:06:46.021733 19963 leader_election.cc:343] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.022249 19963 leader_election.cc:304] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9; no voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d
W20260709 13:06:46.023079 19887 leader_election.cc:343] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.023212 20121 raft_consensus.cc:2749] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:46.025898 20114 tablet_bootstrap.cc:492] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:46.029189 20114 ts_tablet_manager.cc:1403] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.118s	user 0.021s	sys 0.001s
I20260709 13:06:46.031039 20113 tablet_bootstrap.cc:654] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.025761 20119 raft_consensus.cc:493] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.032799 20114 raft_consensus.cc:359] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.033906 20114 raft_consensus.cc:385] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.033849 20119 raft_consensus.cc:515] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:46.034636 20114 raft_consensus.cc:740] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.036165 20114 consensus_queue.cc:260] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.037511 20119 leader_election.cc:290] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.038460 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b051a1e1b9f04842802d2af08b3bbe9b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.039465 19997 raft_consensus.cc:2468] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:46.041348 19887 leader_election.cc:304] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:46.042472 20143 raft_consensus.cc:2804] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.042995 20143 raft_consensus.cc:493] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.043329 20143 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.046870 20119 raft_consensus.cc:2804] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.047405 20119 raft_consensus.cc:493] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.047770 20119 raft_consensus.cc:3060] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.048519 20114 ts_tablet_manager.cc:1434] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.019s	user 0.012s	sys 0.004s
I20260709 13:06:46.050446 20114 tablet_bootstrap.cc:492] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:46.048718 20115 tablet_bootstrap.cc:492] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
W20260709 13:06:46.055253 20036 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.66:35681, user_credentials={real_user=slave}} blocked reactor thread for 45378.5us
I20260709 13:06:46.055476 20115 ts_tablet_manager.cc:1403] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.071s	user 0.011s	sys 0.011s
I20260709 13:06:46.059449 20115 raft_consensus.cc:359] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.060243 20115 raft_consensus.cc:385] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.060539 20115 raft_consensus.cc:740] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.062291 20143 raft_consensus.cc:515] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.063660 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.064415 20120 raft_consensus.cc:515] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.065035 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b051a1e1b9f04842802d2af08b3bbe9b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.065476 20115 consensus_queue.cc:260] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.066774 20120 leader_election.cc:290] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
W20260709 13:06:46.067028 19888 leader_election.cc:343] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.067862 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:46.068518 19997 raft_consensus.cc:3060] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.069116 20113 tablet_bootstrap.cc:492] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:46.069679 20113 ts_tablet_manager.cc:1403] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.081s	user 0.020s	sys 0.009s
I20260709 13:06:46.064411 20072 raft_consensus.cc:2468] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:46.071990 19888 leader_election.cc:304] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 2f3794493e4c4d549f357cd6ea8abbf9
I20260709 13:06:46.072219 20143 leader_election.cc:290] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.072000 20114 tablet_bootstrap.cc:654] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.072911 20113 raft_consensus.cc:359] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.073762 20113 raft_consensus.cc:385] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.073505 20146 raft_consensus.cc:2804] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.074275 20120 raft_consensus.cc:493] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.074309 20146 raft_consensus.cc:493] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.075317 20146 raft_consensus.cc:3060] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.075023 20120 raft_consensus.cc:515] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.081238 19998 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:46.081969 19998 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.084803 20119 raft_consensus.cc:515] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:46.088901 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b051a1e1b9f04842802d2af08b3bbe9b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:46.091629 20146 raft_consensus.cc:515] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
W20260709 13:06:46.092149 19888 leader_election.cc:343] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.092515 20113 raft_consensus.cc:740] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.092672 19997 raft_consensus.cc:2468] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:46.093513 20113 consensus_queue.cc:260] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.094964 19998 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:46.096357 20120 leader_election.cc:290] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.096626 19996 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b051a1e1b9f04842802d2af08b3bbe9b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:46.096812 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:46.097436 19996 raft_consensus.cc:3060] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.097676 19923 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.098553 19995 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
W20260709 13:06:46.100317 20036 leader_election.cc:343] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.102103 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:46.102810 19922 raft_consensus.cc:3060] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.105538 20119 leader_election.cc:290] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Requested vote from peers d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.108058 19996 raft_consensus.cc:2468] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:46.110512 19923 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:46.110818 20113 ts_tablet_manager.cc:1434] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.041s	user 0.006s	sys 0.001s
I20260709 13:06:46.112737 20113 tablet_bootstrap.cc:492] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:46.114122 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.115134 19923 raft_consensus.cc:2393] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d74830b4b12e4c6f938dac068fff593d in current term 1: Already voted for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in this term.
I20260709 13:06:46.115188 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:46.115903 20072 raft_consensus.cc:3060] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.116776 20037 leader_election.cc:304] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.117832 20120 raft_consensus.cc:2804] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.118535 20120 raft_consensus.cc:697] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:46.119469 20120 consensus_queue.cc:237] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.121096 20113 tablet_bootstrap.cc:654] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.121922 19995 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
W20260709 13:06:46.123247 19887 leader_election.cc:343] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.122977 20037 leader_election.cc:304] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d74830b4b12e4c6f938dac068fff593d; no voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:46.126852 20072 raft_consensus.cc:2468] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:46.127856 20147 raft_consensus.cc:2749] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:46.128855 19888 leader_election.cc:304] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 2f3794493e4c4d549f357cd6ea8abbf9
I20260709 13:06:46.127804 20114 tablet_bootstrap.cc:492] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:46.130316 20119 raft_consensus.cc:2804] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.131049 20114 ts_tablet_manager.cc:1403] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.081s	user 0.025s	sys 0.022s
I20260709 13:06:46.129029 20146 leader_election.cc:290] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.132426 19887 leader_election.cc:304] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:46.128906 19922 raft_consensus.cc:2468] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:46.133718 20146 raft_consensus.cc:2804] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.134232 20146 raft_consensus.cc:697] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:46.134730 20113 tablet_bootstrap.cc:492] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:46.134490 20036 leader_election.cc:304] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.135274 20113 ts_tablet_manager.cc:1403] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.007s
I20260709 13:06:46.135478 20147 raft_consensus.cc:2804] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.135134 20146 consensus_queue.cc:237] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:46.136000 20147 raft_consensus.cc:697] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:46.136443 20114 raft_consensus.cc:359] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.137245 20114 raft_consensus.cc:385] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.137624 20114 raft_consensus.cc:740] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.136919 20147 consensus_queue.cc:237] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.138521 20114 consensus_queue.cc:260] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.138813 20113 raft_consensus.cc:359] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:46.139678 20113 raft_consensus.cc:385] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.140221 20113 raft_consensus.cc:740] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.140933 20113 consensus_queue.cc:260] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:46.143474 20113 ts_tablet_manager.cc:1434] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 13:06:46.144734 20113 tablet_bootstrap.cc:492] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:46.139560 20115 ts_tablet_manager.cc:1434] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.083s	user 0.007s	sys 0.000s
I20260709 13:06:46.150888 20119 raft_consensus.cc:697] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:46.151279 20114 ts_tablet_manager.cc:1434] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.005s
I20260709 13:06:46.152442 20114 tablet_bootstrap.cc:492] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:46.152002 20119 consensus_queue.cc:237] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.155603 19820 catalog_manager.cc:5697] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 0 to 1, leader changed from <none> to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 1 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:46.160338 20115 tablet_bootstrap.cc:492] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:46.161744 20114 tablet_bootstrap.cc:654] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.165767 20121 raft_consensus.cc:493] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.166487 20121 raft_consensus.cc:515] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.168826 20121 leader_election.cc:290] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.171413 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cbe74d298c4e6c848fab0a23ed0917" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.172154 20072 raft_consensus.cc:2468] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:46.172816 20115 tablet_bootstrap.cc:654] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.173725 19963 leader_election.cc:304] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.174777 20121 raft_consensus.cc:2804] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.175208 20121 raft_consensus.cc:493] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.172279 20113 tablet_bootstrap.cc:654] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.175618 20121 raft_consensus.cc:3060] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.183137 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cbe74d298c4e6c848fab0a23ed0917" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
W20260709 13:06:46.185245 19963 leader_election.cc:343] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.197773 20121 raft_consensus.cc:515] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.211498 20114 tablet_bootstrap.cc:492] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:46.212138 20114 ts_tablet_manager.cc:1403] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.060s	user 0.013s	sys 0.032s
I20260709 13:06:46.212312 20113 tablet_bootstrap.cc:492] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:46.213275 20113 ts_tablet_manager.cc:1403] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.069s	user 0.015s	sys 0.016s
I20260709 13:06:46.213742 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cbe74d298c4e6c848fab0a23ed0917" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:46.215808 20114 raft_consensus.cc:359] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.216552 20114 raft_consensus.cc:385] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.216620 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cbe74d298c4e6c848fab0a23ed0917" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:46.216898 20114 raft_consensus.cc:740] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.217084 20113 raft_consensus.cc:359] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.218155 20113 raft_consensus.cc:385] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:06:46.218182 19963 leader_election.cc:343] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.219362 20113 raft_consensus.cc:740] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.217371 19821 catalog_manager.cc:5697] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:46.217381 20072 raft_consensus.cc:3060] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.218003 20114 consensus_queue.cc:260] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.220242 20113 consensus_queue.cc:260] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.222374 20114 ts_tablet_manager.cc:1434] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.004s
I20260709 13:06:46.223057 20113 ts_tablet_manager.cc:1434] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 13:06:46.223791 20114 tablet_bootstrap.cc:492] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:46.224187 20113 tablet_bootstrap.cc:492] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:46.230022 20072 raft_consensus.cc:2468] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:46.231600 19963 leader_election.cc:304] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:46.231974 20113 tablet_bootstrap.cc:654] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.231974 20114 tablet_bootstrap.cc:654] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.233857 20162 raft_consensus.cc:2804] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.218227 20121 leader_election.cc:290] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.242772 20113 tablet_bootstrap.cc:492] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:46.242918 20115 tablet_bootstrap.cc:492] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:46.243366 20115 ts_tablet_manager.cc:1403] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.083s	user 0.009s	sys 0.014s
I20260709 13:06:46.243393 20113 ts_tablet_manager.cc:1403] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.006s
I20260709 13:06:46.244006 20114 tablet_bootstrap.cc:492] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:46.244557 20114 ts_tablet_manager.cc:1403] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.007s
I20260709 13:06:46.246196 20119 raft_consensus.cc:493] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.246347 20113 raft_consensus.cc:359] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.247053 20113 raft_consensus.cc:385] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.246724 20119 raft_consensus.cc:515] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.248931 20119 leader_election.cc:290] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.248680 20114 raft_consensus.cc:359] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.249624 20114 raft_consensus.cc:385] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.250008 20114 raft_consensus.cc:740] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.250679 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cbe74d298c4e6c848fab0a23ed0917" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.251160 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cbe74d298c4e6c848fab0a23ed0917" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.251509 20072 raft_consensus.cc:2393] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in current term 1: Already voted for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in this term.
I20260709 13:06:46.251847 20113 raft_consensus.cc:740] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.250902 20114 consensus_queue.cc:260] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.252848 20113 consensus_queue.cc:260] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.254875 20114 ts_tablet_manager.cc:1434] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:06:46.255478 20113 ts_tablet_manager.cc:1434] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:06:46.255991 20114 tablet_bootstrap.cc:492] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:46.256570 20113 tablet_bootstrap.cc:492] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:46.256554 20162 raft_consensus.cc:697] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:46.257716 20162 consensus_queue.cc:237] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.263096 20113 tablet_bootstrap.cc:654] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.263096 20114 tablet_bootstrap.cc:654] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.266180 19820 catalog_manager.cc:5697] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 0 to 1, leader changed from <none> to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 1 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:46.249701 20115 raft_consensus.cc:359] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.274653 20115 raft_consensus.cc:385] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.275027 20147 raft_consensus.cc:493] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.275843 20147 raft_consensus.cc:515] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.276950 19818 catalog_manager.cc:5697] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:46.279661 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.279541 19887 leader_election.cc:304] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d
W20260709 13:06:46.281670 20037 leader_election.cc:343] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.281817 20119 raft_consensus.cc:3060] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.285480 20114 tablet_bootstrap.cc:492] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:46.286092 20114 ts_tablet_manager.cc:1403] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.030s	user 0.014s	sys 0.010s
I20260709 13:06:46.289371 20114 raft_consensus.cc:359] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.290094 20114 raft_consensus.cc:385] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.290378 20114 raft_consensus.cc:740] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.291047 20114 consensus_queue.cc:260] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.291425 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.294198 20114 ts_tablet_manager.cc:1434] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 13:06:46.294359 19821 catalog_manager.cc:5697] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:46.296080 20036 leader_election.cc:343] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:46.296682 20036 leader_election.cc:304] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d74830b4b12e4c6f938dac068fff593d; no voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:46.275032 20115 raft_consensus.cc:740] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.297447 20120 raft_consensus.cc:2749] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:46.297556 20147 leader_election.cc:290] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.298600 20115 consensus_queue.cc:260] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.299647 20119 raft_consensus.cc:2749] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:46.301085 20115 ts_tablet_manager.cc:1434] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.057s	user 0.005s	sys 0.002s
I20260709 13:06:46.302075 20115 tablet_bootstrap.cc:492] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:46.302675 20113 tablet_bootstrap.cc:492] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:46.303224 20113 ts_tablet_manager.cc:1403] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.047s	user 0.012s	sys 0.007s
I20260709 13:06:46.306907 20113 raft_consensus.cc:359] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.307701 20113 raft_consensus.cc:385] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.308018 20113 raft_consensus.cc:740] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.308557 20115 tablet_bootstrap.cc:654] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.308945 20113 consensus_queue.cc:260] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.311870 20113 ts_tablet_manager.cc:1434] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 13:06:46.315374 20115 tablet_bootstrap.cc:492] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:46.315976 20115 ts_tablet_manager.cc:1403] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.005s
I20260709 13:06:46.318876 20115 raft_consensus.cc:359] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.319624 20115 raft_consensus.cc:385] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.319938 20115 raft_consensus.cc:740] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.320690 20115 consensus_queue.cc:260] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.321127 20162 raft_consensus.cc:493] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.321755 20162 raft_consensus.cc:515] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.323011 20115 ts_tablet_manager.cc:1434] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:06:46.323980 20162 leader_election.cc:290] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.324131 20115 tablet_bootstrap.cc:492] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:46.327091 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c113c7eeba4c09ab636538d2add58a" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.327906 19922 raft_consensus.cc:2468] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:46.329308 19963 leader_election.cc:304] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:46.329846 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c113c7eeba4c09ab636538d2add58a" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.330624 20072 raft_consensus.cc:2468] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:46.330933 20115 tablet_bootstrap.cc:654] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:46.332604 20162 raft_consensus.cc:2804] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.333081 20162 raft_consensus.cc:493] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.333475 20162 raft_consensus.cc:3060] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.336230 20115 tablet_bootstrap.cc:492] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:46.336843 20115 ts_tablet_manager.cc:1403] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 13:06:46.340009 20115 raft_consensus.cc:359] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.340881 20115 raft_consensus.cc:385] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:46.341104 20115 raft_consensus.cc:740] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:46.340946 20162 raft_consensus.cc:515] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.342073 20115 consensus_queue.cc:260] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.343277 20162 leader_election.cc:290] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.344276 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c113c7eeba4c09ab636538d2add58a" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:46.344573 20115 ts_tablet_manager.cc:1434] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 13:06:46.344514 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c113c7eeba4c09ab636538d2add58a" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:46.344967 19922 raft_consensus.cc:3060] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.345252 20072 raft_consensus.cc:3060] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.349583 20119 raft_consensus.cc:493] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.350637 20119 raft_consensus.cc:515] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.352589 19922 raft_consensus.cc:2468] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:46.353500 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.352589 20072 raft_consensus.cc:2468] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:46.353945 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.354353 19997 raft_consensus.cc:2468] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:46.354734 20071 raft_consensus.cc:2468] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:46.355360 19963 leader_election.cc:304] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:46.355611 19887 leader_election.cc:304] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:46.356793 20146 raft_consensus.cc:2804] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.356967 20119 leader_election.cc:290] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.357164 20146 raft_consensus.cc:493] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.357715 20146 raft_consensus.cc:3060] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.360637 20162 raft_consensus.cc:2804] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.361220 20162 raft_consensus.cc:697] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:46.362519 20162 consensus_queue.cc:237] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.362897 20146 raft_consensus.cc:515] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.365664 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:46.366325 19997 raft_consensus.cc:3060] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.366081 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:46.367144 20071 raft_consensus.cc:3060] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.369078 20146 leader_election.cc:290] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.375825 20071 raft_consensus.cc:2468] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:46.374842 19821 catalog_manager.cc:5697] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:46.376663 19997 raft_consensus.cc:2468] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:46.377065 19888 leader_election.cc:304] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.377992 20146 raft_consensus.cc:2804] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.378463 20146 raft_consensus.cc:697] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:46.379323 20146 consensus_queue.cc:237] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.382566 20147 raft_consensus.cc:493] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.383245 20147 raft_consensus.cc:515] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.388429 19820 catalog_manager.cc:5697] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:46.389756 20147 leader_election.cc:290] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.390347 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.390780 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.391001 19922 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:46.391574 19997 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:46.392728 20036 leader_election.cc:304] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.393949 20147 raft_consensus.cc:2804] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.394304 20147 raft_consensus.cc:493] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.394629 20147 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.402545 20147 raft_consensus.cc:515] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.404613 20147 leader_election.cc:290] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.405705 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:46.405969 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:46.406380 19922 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.406755 19997 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.410562 20146 raft_consensus.cc:493] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.411190 20146 raft_consensus.cc:515] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.412788 19922 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:46.412935 19997 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:46.413895 20146 leader_election.cc:290] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.414398 20037 leader_election.cc:304] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.415084 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.415202 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.415485 20147 raft_consensus.cc:2804] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.415875 20071 raft_consensus.cc:2468] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:46.416167 20147 raft_consensus.cc:697] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:46.416082 19997 raft_consensus.cc:2468] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:46.417093 19888 leader_election.cc:304] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.417148 20147 consensus_queue.cc:237] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.418305 20146 raft_consensus.cc:2804] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:46.418684 20146 raft_consensus.cc:493] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:46.419029 20146 raft_consensus.cc:3060] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.425048 20146 raft_consensus.cc:515] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.427287 20146 leader_election.cc:290] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.426576 19820 catalog_manager.cc:5697] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 0 to 1, leader changed from <none> to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 1 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:46.427980 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:46.427953 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:46.428696 19997 raft_consensus.cc:3060] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.428735 20071 raft_consensus.cc:3060] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.435333 20071 raft_consensus.cc:2468] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:46.435550 19997 raft_consensus.cc:2468] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:46.436605 19888 leader_election.cc:304] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.437484 20146 raft_consensus.cc:2804] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:46.437968 20146 raft_consensus.cc:697] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:46.438781 20146 consensus_queue.cc:237] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.446416 19820 catalog_manager.cc:5697] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:46.498342 20162 raft_consensus.cc:493] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.498855 20162 raft_consensus.cc:515] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.500612 20162 leader_election.cc:290] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.500775 20147 raft_consensus.cc:493] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:46.501348 20147 raft_consensus.cc:515] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } } peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } }
I20260709 13:06:46.501986 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.502033 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2c7846e41f249af96a83b97e089f89f" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:46.503182 20071 raft_consensus.cc:2393] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f3794493e4c4d549f357cd6ea8abbf9 in current term 1: Already voted for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in this term.
I20260709 13:06:46.504315 19963 leader_election.cc:304] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9; no voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:46.504433 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b051a1e1b9f04842802d2af08b3bbe9b" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:46.505312 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b051a1e1b9f04842802d2af08b3bbe9b" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:46.505802 20162 raft_consensus.cc:3060] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.506150 19997 raft_consensus.cc:2393] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d74830b4b12e4c6f938dac068fff593d in current term 1: Already voted for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in this term.
I20260709 13:06:46.504683 20147 leader_election.cc:290] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.507323 20036 leader_election.cc:304] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d74830b4b12e4c6f938dac068fff593d; no voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:46.508132 20147 raft_consensus.cc:3060] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:46.511803 20162 raft_consensus.cc:2749] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:46.513794 20147 raft_consensus.cc:2749] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:46.580849 20146 consensus_queue.cc:1048] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.593832 20152 consensus_queue.cc:1048] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.609158 20146 consensus_queue.cc:1048] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.611521 20147 consensus_queue.cc:1048] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.623353 20119 consensus_queue.cc:1048] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.637048 20143 consensus_queue.cc:1048] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.649746 20143 consensus_queue.cc:1048] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.665258 20071 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
mode: GRACEFUL
new_leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:40592
I20260709 13:06:46.666148 20071 raft_consensus.cc:606] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Received request to transfer leadership to TS 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:46.667138 20147 consensus_queue.cc:1048] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.681800 20071 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
mode: GRACEFUL
new_leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:40592
I20260709 13:06:46.682472 20071 raft_consensus.cc:606] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Received request to transfer leadership to TS 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:46.689000 20071 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
mode: GRACEFUL
new_leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
 from {username='slave'} at 127.0.0.1:40592
I20260709 13:06:46.689636 20071 raft_consensus.cc:606] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Received request to transfer leadership to TS 2f3794493e4c4d549f357cd6ea8abbf9
I20260709 13:06:46.712950 19923 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5"
dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
mode: GRACEFUL
new_leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
 from {username='slave'} at 127.0.0.1:50078
I20260709 13:06:46.713550 19923 raft_consensus.cc:606] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Received request to transfer leadership to TS 2f3794493e4c4d549f357cd6ea8abbf9
I20260709 13:06:46.719815 19923 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69"
dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
mode: GRACEFUL
new_leader_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:50078
I20260709 13:06:46.720374 19923 raft_consensus.cc:606] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Received request to transfer leadership to TS d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:46.753798 20162 consensus_queue.cc:1048] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.766813 20191 consensus_queue.cc:1048] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.788796 20191 consensus_queue.cc:1048] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.800252 20162 consensus_queue.cc:1048] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.874195 20146 consensus_queue.cc:1048] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.887394 20152 consensus_queue.cc:1048] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.896422 20146 raft_consensus.cc:993] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1: : Instructing follower d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:46.896986 20152 raft_consensus.cc:1081] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Signalling peer d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:46.899005 20071 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:40586
I20260709 13:06:46.899717 20071 raft_consensus.cc:493] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:46.900133 20071 raft_consensus.cc:3060] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:46.906184 20071 raft_consensus.cc:515] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.908205 20071 leader_election.cc:290] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 2 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:46.909093 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:46.909538 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18f26dbbb33b4bc096697663cb163f69" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:46.909883 19923 raft_consensus.cc:3055] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:46.910310 19923 raft_consensus.cc:740] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:46.910342 19997 raft_consensus.cc:3060] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:46.911324 19923 consensus_queue.cc:260] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.912762 19923 raft_consensus.cc:3060] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:46.917711 19997 raft_consensus.cc:2468] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 2.
I20260709 13:06:46.918898 20036 leader_election.cc:304] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.919566 19923 raft_consensus.cc:2468] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 2.
I20260709 13:06:46.919754 20143 raft_consensus.cc:2804] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:46.920379 20143 raft_consensus.cc:697] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 2 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:46.921240 20143 consensus_queue.cc:237] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.928217 19818 catalog_manager.cc:5697] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 1 to 2, leader changed from 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65) to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 2 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:46.942086 20152 consensus_queue.cc:1048] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.949134 20146 raft_consensus.cc:993] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1: : Instructing follower 2f3794493e4c4d549f357cd6ea8abbf9 to start an election
I20260709 13:06:46.949781 20151 raft_consensus.cc:1081] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Signalling peer 2f3794493e4c4d549f357cd6ea8abbf9 to start an election
I20260709 13:06:46.951534 19998 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5"
dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
 from {username='slave'} at 127.0.0.1:49524
I20260709 13:06:46.954758 19998 raft_consensus.cc:493] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:46.955246 19998 raft_consensus.cc:3060] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:46.955286 20151 consensus_queue.cc:1048] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.957016 20120 consensus_queue.cc:1048] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:46.964049 19998 raft_consensus.cc:515] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.971498 19998 leader_election.cc:290] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 2 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:46.975440 20147 raft_consensus.cc:993] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d: : Instructing follower 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:46.975968 20147 raft_consensus.cc:1081] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Signalling peer 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:46.979229 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:46.979766 19921 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1"
dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:50066
I20260709 13:06:46.980087 20071 raft_consensus.cc:3060] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:46.980497 19921 raft_consensus.cc:493] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:46.980901 19921 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:46.977680 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32b285df036b4239b948bab122c4b8a5" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:46.981878 20147 consensus_queue.cc:1048] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:46.982168 19923 raft_consensus.cc:3055] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:46.983742 19923 raft_consensus.cc:740] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:46.987951 19923 consensus_queue.cc:260] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.989689 19923 raft_consensus.cc:3060] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:46.992043 20071 raft_consensus.cc:2468] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 2.
I20260709 13:06:46.993884 19963 leader_election.cc:304] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:46.995304 20162 raft_consensus.cc:2804] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:46.996052 20162 raft_consensus.cc:697] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:46.997300 20162 consensus_queue.cc:237] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:46.998384 19923 raft_consensus.cc:2468] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 2.
I20260709 13:06:47.000669 19921 raft_consensus.cc:515] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.004240 19921 leader_election.cc:290] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:47.006052 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:47.006812 19997 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.007294 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:47.008077 20071 raft_consensus.cc:3055] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:47.010105 20071 raft_consensus.cc:740] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:47.014371 20071 consensus_queue.cc:260] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.015414 19997 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:47.015959 20071 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.016768 19887 leader_election.cc:304] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:47.017717 20152 raft_consensus.cc:2804] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.018239 20152 raft_consensus.cc:697] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:47.019021 20152 consensus_queue.cc:237] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.023416 20071 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:47.023298 19820 catalog_manager.cc:5697] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 1 to 2, leader changed from 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65) to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.033871 19818 catalog_manager.cc:5697] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 1 to 2, leader changed from d74830b4b12e4c6f938dac068fff593d (127.7.78.67) to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.055619 20120 raft_consensus.cc:993] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d: : Instructing follower 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:47.056267 20147 raft_consensus.cc:1081] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Signalling peer 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:47.058068 19921 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338"
dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:50066
I20260709 13:06:47.058665 19921 raft_consensus.cc:493] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:47.059057 19921 raft_consensus.cc:3060] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.065227 19921 raft_consensus.cc:515] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.067229 19921 leader_election.cc:290] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:47.068151 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:47.068387 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51d91599e9b540e3a3c7a75ea0d5a338" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:47.068941 19997 raft_consensus.cc:3060] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.069113 20072 raft_consensus.cc:3055] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:47.069623 20072 raft_consensus.cc:740] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:47.070416 20072 consensus_queue.cc:260] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.071587 20072 raft_consensus.cc:3060] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.074749 19997 raft_consensus.cc:2468] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:47.076156 19887 leader_election.cc:304] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:47.077193 20152 raft_consensus.cc:2804] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.077761 20152 raft_consensus.cc:697] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:47.078559 20072 raft_consensus.cc:2468] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:47.078727 20152 consensus_queue.cc:237] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.086817 19818 catalog_manager.cc:5697] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 1 to 2, leader changed from d74830b4b12e4c6f938dac068fff593d (127.7.78.67) to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.216552 20120 raft_consensus.cc:993] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d: : Instructing follower 2f3794493e4c4d549f357cd6ea8abbf9 to start an election
I20260709 13:06:47.217156 20147 raft_consensus.cc:1081] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Signalling peer 2f3794493e4c4d549f357cd6ea8abbf9 to start an election
I20260709 13:06:47.218840 19997 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336"
dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
 from {username='slave'} at 127.0.0.1:49520
I20260709 13:06:47.219470 19997 raft_consensus.cc:493] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:47.219938 19997 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.228598 19997 raft_consensus.cc:515] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.230996 19997 leader_election.cc:290] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 2 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:47.231920 19921 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:47.232136 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:47.232735 19921 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.232800 20072 raft_consensus.cc:3055] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:47.233351 20072 raft_consensus.cc:740] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:47.234393 20072 consensus_queue.cc:260] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.235874 20072 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.241637 19921 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 2.
I20260709 13:06:47.243261 19963 leader_election.cc:304] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:47.244266 20072 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 2.
I20260709 13:06:47.244342 20162 raft_consensus.cc:2804] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.245131 20162 raft_consensus.cc:697] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:47.246094 20162 consensus_queue.cc:237] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:47.253186 19820 catalog_manager.cc:5697] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 1 to 2, leader changed from d74830b4b12e4c6f938dac068fff593d (127.7.78.67) to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 2 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.398895 19921 raft_consensus.cc:1275] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.400089 20147 consensus_queue.cc:1048] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:47.407862 19997 raft_consensus.cc:1275] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.409926 20147 consensus_queue.cc:1048] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:47.504356 20072 raft_consensus.cc:1275] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.506006 20151 consensus_queue.cc:1048] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:47.514637 19997 raft_consensus.cc:1275] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.516970 20119 consensus_queue.cc:1048] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:47.535151 20072 raft_consensus.cc:1275] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.536386 20119 consensus_queue.cc:1048] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:47.545152 19997 raft_consensus.cc:1275] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.547127 20201 consensus_queue.cc:1048] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:47.579778 19921 raft_consensus.cc:1275] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Refusing update from remote peer 2f3794493e4c4d549f357cd6ea8abbf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.581172 20162 consensus_queue.cc:1048] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:47.589967 20072 raft_consensus.cc:1275] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Refusing update from remote peer 2f3794493e4c4d549f357cd6ea8abbf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.592123 20163 consensus_queue.cc:1048] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:47.646142 19921 raft_consensus.cc:1275] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Refusing update from remote peer 2f3794493e4c4d549f357cd6ea8abbf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.647723 20163 consensus_queue.cc:1048] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:47.655774 20072 raft_consensus.cc:1275] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Refusing update from remote peer 2f3794493e4c4d549f357cd6ea8abbf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.657230 20191 consensus_queue.cc:1048] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:49.755995 19820 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50504:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:49.759603 19820 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:49.811372 19906 tablet_service.cc:1511] Processing CreateTablet for tablet 085ced9b763c4a69a5f0f16de3a724a7 (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:49.812703 19906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 085ced9b763c4a69a5f0f16de3a724a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.812351 19986 tablet_service.cc:1511] Processing CreateTablet for tablet 085ced9b763c4a69a5f0f16de3a724a7 (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:49.813833 19986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 085ced9b763c4a69a5f0f16de3a724a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.814308 20056 tablet_service.cc:1511] Processing CreateTablet for tablet 085ced9b763c4a69a5f0f16de3a724a7 (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:06:49.815856 20056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 085ced9b763c4a69a5f0f16de3a724a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.816442 19987 tablet_service.cc:1511] Processing CreateTablet for tablet bda4eae76b144b33836571c87c8bc0e7 (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:49.817920 19987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda4eae76b144b33836571c87c8bc0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.821247 19907 tablet_service.cc:1511] Processing CreateTablet for tablet bda4eae76b144b33836571c87c8bc0e7 (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:49.822645 19907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda4eae76b144b33836571c87c8bc0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.827869 19988 tablet_service.cc:1511] Processing CreateTablet for tablet a398a0d465a3459bb48f3564e5a0b87b (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:49.828181 20059 tablet_service.cc:1511] Processing CreateTablet for tablet bda4eae76b144b33836571c87c8bc0e7 (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:06:49.829327 19988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a398a0d465a3459bb48f3564e5a0b87b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.829582 20059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda4eae76b144b33836571c87c8bc0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.834509 19908 tablet_service.cc:1511] Processing CreateTablet for tablet a398a0d465a3459bb48f3564e5a0b87b (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:49.835922 19908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a398a0d465a3459bb48f3564e5a0b87b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.838663 20054 tablet_service.cc:1511] Processing CreateTablet for tablet a398a0d465a3459bb48f3564e5a0b87b (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:06:49.840132 20054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a398a0d465a3459bb48f3564e5a0b87b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.846606 19905 tablet_service.cc:1511] Processing CreateTablet for tablet 6752ecec869a49b79b3c29f06baab9ee (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:49.848088 19905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6752ecec869a49b79b3c29f06baab9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.865011 19988 tablet_service.cc:1511] Processing CreateTablet for tablet 6752ecec869a49b79b3c29f06baab9ee (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:49.866503 19988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6752ecec869a49b79b3c29f06baab9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.867120 20062 tablet_service.cc:1511] Processing CreateTablet for tablet 6752ecec869a49b79b3c29f06baab9ee (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:06:49.869026 20062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6752ecec869a49b79b3c29f06baab9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.882520 20218 tablet_bootstrap.cc:492] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:49.886948 19911 tablet_service.cc:1511] Processing CreateTablet for tablet 8415f954b8144792a2ac934b00f876aa (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:49.888068 19986 tablet_service.cc:1511] Processing CreateTablet for tablet 8415f954b8144792a2ac934b00f876aa (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:49.888455 19911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8415f954b8144792a2ac934b00f876aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.890858 20061 tablet_service.cc:1511] Processing CreateTablet for tablet 8415f954b8144792a2ac934b00f876aa (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:06:49.892235 20061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8415f954b8144792a2ac934b00f876aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.898298 20218 tablet_bootstrap.cc:654] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:49.900372 19987 tablet_service.cc:1511] Processing CreateTablet for tablet de5a97d598a3401da6fdaac95f8c5d0c (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:49.901893 19987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5a97d598a3401da6fdaac95f8c5d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.906005 20219 tablet_bootstrap.cc:492] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:49.918123 19986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8415f954b8144792a2ac934b00f876aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.929829 20055 tablet_service.cc:1511] Processing CreateTablet for tablet de5a97d598a3401da6fdaac95f8c5d0c (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:49.931337 20055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5a97d598a3401da6fdaac95f8c5d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.932344 19906 tablet_service.cc:1511] Processing CreateTablet for tablet 92383a893f2a4ccfa25834354dd990bf (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:49.933748 19906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92383a893f2a4ccfa25834354dd990bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.938336 20221 tablet_bootstrap.cc:492] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
W20260709 13:06:49.939574 19869 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:49.939402 19982 tablet_service.cc:1511] Processing CreateTablet for tablet 92383a893f2a4ccfa25834354dd990bf (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:49.940765 19982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92383a893f2a4ccfa25834354dd990bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.941751 19907 tablet_service.cc:1511] Processing CreateTablet for tablet de5a97d598a3401da6fdaac95f8c5d0c (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:06:49.943218 19907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5a97d598a3401da6fdaac95f8c5d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.943382 20219 tablet_bootstrap.cc:654] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:49.951745 20218 tablet_bootstrap.cc:492] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:49.952364 20218 ts_tablet_manager.cc:1403] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.070s	user 0.009s	sys 0.028s
I20260709 13:06:49.953420 19988 tablet_service.cc:1511] Processing CreateTablet for tablet d657a45cf2fe4cb592aa1b9c132b519e (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:49.954794 19988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d657a45cf2fe4cb592aa1b9c132b519e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.955695 20218 raft_consensus.cc:359] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:49.956483 19905 tablet_service.cc:1511] Processing CreateTablet for tablet 0f6f7e01e1024f098867a2e2652114bc (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:49.957183 19983 tablet_service.cc:1511] Processing CreateTablet for tablet 0f6f7e01e1024f098867a2e2652114bc (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:49.957631 20218 raft_consensus.cc:385] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:49.958120 20218 raft_consensus.cc:740] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:49.958313 19905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f6f7e01e1024f098867a2e2652114bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.960227 19911 tablet_service.cc:1511] Processing CreateTablet for tablet d657a45cf2fe4cb592aa1b9c132b519e (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:49.958598 20059 tablet_service.cc:1511] Processing CreateTablet for tablet d657a45cf2fe4cb592aa1b9c132b519e (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:06:49.961769 20059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d657a45cf2fe4cb592aa1b9c132b519e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.958534 19983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f6f7e01e1024f098867a2e2652114bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.960685 20221 tablet_bootstrap.cc:654] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:49.966216 20218 consensus_queue.cc:260] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:49.968752 20218 ts_tablet_manager.cc:1434] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20260709 13:06:49.959437 20054 tablet_service.cc:1511] Processing CreateTablet for tablet 92383a893f2a4ccfa25834354dd990bf (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:06:49.982847 19911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d657a45cf2fe4cb592aa1b9c132b519e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:49.985879 20054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92383a893f2a4ccfa25834354dd990bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:50.000897 20056 tablet_service.cc:1511] Processing CreateTablet for tablet 0f6f7e01e1024f098867a2e2652114bc (DEFAULT_TABLE table=soft_deleted_table [id=252486229c564a4891c8ba398528c33c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:06:50.002540 20056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f6f7e01e1024f098867a2e2652114bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:50.006531 20219 tablet_bootstrap.cc:492] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.007215 20219 ts_tablet_manager.cc:1403] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.102s	user 0.008s	sys 0.008s
W20260709 13:06:50.007468 20024 tablet.cc:2406] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:06:50.010603 20219 raft_consensus.cc:359] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.011471 20219 raft_consensus.cc:385] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.011756 20219 raft_consensus.cc:740] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.012651 20219 consensus_queue.cc:260] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.014261 20218 tablet_bootstrap.cc:492] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.020566 20219 ts_tablet_manager.cc:1434] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.002s
I20260709 13:06:50.046195 20219 tablet_bootstrap.cc:492] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.048257 20221 tablet_bootstrap.cc:492] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.048997 20221 ts_tablet_manager.cc:1403] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.111s	user 0.007s	sys 0.008s
I20260709 13:06:50.051429 20218 tablet_bootstrap.cc:654] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.053068 20219 tablet_bootstrap.cc:654] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.065918 20221 raft_consensus.cc:359] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.066977 20221 raft_consensus.cc:385] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.067484 20221 raft_consensus.cc:740] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.068622 20221 consensus_queue.cc:260] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.078004 20221 ts_tablet_manager.cc:1434] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.001s
I20260709 13:06:50.079773 20221 tablet_bootstrap.cc:492] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.084416 20219 tablet_bootstrap.cc:492] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.085099 20219 ts_tablet_manager.cc:1403] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.015s
I20260709 13:06:50.086261 20218 tablet_bootstrap.cc:492] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.087110 20218 ts_tablet_manager.cc:1403] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.073s	user 0.013s	sys 0.004s
I20260709 13:06:50.088207 20221 tablet_bootstrap.cc:654] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.088759 20219 raft_consensus.cc:359] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.091944 20219 raft_consensus.cc:385] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.092456 20219 raft_consensus.cc:740] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.090834 20218 raft_consensus.cc:359] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.093686 20218 raft_consensus.cc:385] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.094142 20218 raft_consensus.cc:740] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.095443 20219 consensus_queue.cc:260] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.098305 20219 ts_tablet_manager.cc:1434] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:06:50.099532 20219 tablet_bootstrap.cc:492] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.100871 20221 tablet_bootstrap.cc:492] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.095098 20218 consensus_queue.cc:260] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.101843 20221 ts_tablet_manager.cc:1403] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.022s	user 0.016s	sys 0.004s
I20260709 13:06:50.103371 20218 ts_tablet_manager.cc:1434] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 13:06:50.105227 20221 raft_consensus.cc:359] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.106097 20221 raft_consensus.cc:385] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.106403 20221 raft_consensus.cc:740] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.107143 20221 consensus_queue.cc:260] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.107800 20219 tablet_bootstrap.cc:654] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.109582 20221 ts_tablet_manager.cc:1434] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 13:06:50.110672 20221 tablet_bootstrap.cc:492] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.114856 20218 tablet_bootstrap.cc:492] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.116364 20219 tablet_bootstrap.cc:492] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.116982 20219 ts_tablet_manager.cc:1403] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.002s
I20260709 13:06:50.121932 20219 raft_consensus.cc:359] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.122574 20218 tablet_bootstrap.cc:654] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.122785 20219 raft_consensus.cc:385] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.123157 20219 raft_consensus.cc:740] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.123380 20221 tablet_bootstrap.cc:654] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.124123 20219 consensus_queue.cc:260] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.127211 20219 ts_tablet_manager.cc:1434] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.005s
I20260709 13:06:50.128629 20219 tablet_bootstrap.cc:492] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.136574 20218 tablet_bootstrap.cc:492] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.137264 20218 ts_tablet_manager.cc:1403] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.023s	user 0.008s	sys 0.007s
I20260709 13:06:50.138309 20219 tablet_bootstrap.cc:654] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.140616 20221 tablet_bootstrap.cc:492] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.141229 20221 ts_tablet_manager.cc:1403] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.031s	user 0.007s	sys 0.011s
I20260709 13:06:50.142177 20218 raft_consensus.cc:359] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.142971 20218 raft_consensus.cc:385] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.143225 20218 raft_consensus.cc:740] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.143937 20218 consensus_queue.cc:260] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.146497 20218 ts_tablet_manager.cc:1434] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 13:06:50.148114 20218 tablet_bootstrap.cc:492] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.148341 20221 raft_consensus.cc:359] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.149706 20219 tablet_bootstrap.cc:492] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.150354 20219 ts_tablet_manager.cc:1403] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.022s	user 0.017s	sys 0.004s
I20260709 13:06:50.152714 20219 raft_consensus.cc:359] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.149765 20221 raft_consensus.cc:385] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.153431 20219 raft_consensus.cc:385] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.153916 20221 raft_consensus.cc:740] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.153980 20219 raft_consensus.cc:740] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.154822 20221 consensus_queue.cc:260] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.154866 20219 consensus_queue.cc:260] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.155519 20218 tablet_bootstrap.cc:654] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.157838 20219 ts_tablet_manager.cc:1434] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:06:50.158288 20221 ts_tablet_manager.cc:1434] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:06:50.158878 20219 tablet_bootstrap.cc:492] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.166251 20219 tablet_bootstrap.cc:654] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.159750 20221 tablet_bootstrap.cc:492] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.188887 20221 tablet_bootstrap.cc:654] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.193995 20218 tablet_bootstrap.cc:492] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.194590 20218 ts_tablet_manager.cc:1403] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.047s	user 0.023s	sys 0.021s
I20260709 13:06:50.194940 20219 tablet_bootstrap.cc:492] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.195286 20221 tablet_bootstrap.cc:492] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.195731 20219 ts_tablet_manager.cc:1403] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.037s	user 0.020s	sys 0.015s
I20260709 13:06:50.195875 20221 ts_tablet_manager.cc:1403] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.036s	user 0.015s	sys 0.000s
I20260709 13:06:50.197578 20205 raft_consensus.cc:493] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.197713 20218 raft_consensus.cc:359] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.198159 20205 raft_consensus.cc:515] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.198637 20218 raft_consensus.cc:385] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.198995 20218 raft_consensus.cc:740] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.199077 20219 raft_consensus.cc:359] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.199262 20221 raft_consensus.cc:359] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.200340 20219 raft_consensus.cc:385] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.199950 20218 consensus_queue.cc:260] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.200609 20221 raft_consensus.cc:385] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.201467 20219 raft_consensus.cc:740] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.203114 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:50.203002 20219 consensus_queue.cc:260] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.203670 20218 ts_tablet_manager.cc:1434] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.002s
I20260709 13:06:50.204530 20221 raft_consensus.cc:740] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.204849 20218 tablet_bootstrap.cc:492] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.204967 19921 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:50.205840 19921 raft_consensus.cc:2468] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:50.205617 20221 consensus_queue.cc:260] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.206214 20219 ts_tablet_manager.cc:1434] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:06:50.207134 20227 raft_consensus.cc:493] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.207630 20227 raft_consensus.cc:515] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.209602 20219 tablet_bootstrap.cc:492] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.210863 20037 leader_election.cc:304] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:50.213601 20218 tablet_bootstrap.cc:654] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.220012 20227 leader_election.cc:290] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.221822 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752ecec869a49b79b3c29f06baab9ee" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.222388 20219 tablet_bootstrap.cc:654] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.222581 20072 raft_consensus.cc:2468] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:50.225088 19888 leader_election.cc:304] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:50.229076 19998 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752ecec869a49b79b3c29f06baab9ee" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:50.230186 19998 raft_consensus.cc:2468] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:50.230434 20218 tablet_bootstrap.cc:492] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.231333 20218 ts_tablet_manager.cc:1403] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.027s	user 0.002s	sys 0.020s
I20260709 13:06:50.237179 20218 raft_consensus.cc:359] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.238432 20218 raft_consensus.cc:385] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.238842 20213 raft_consensus.cc:493] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.240187 20213 raft_consensus.cc:515] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.203938 19997 raft_consensus.cc:2468] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 0.
I20260709 13:06:50.241582 20205 leader_election.cc:290] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:50.242797 20205 raft_consensus.cc:2804] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.243238 20205 raft_consensus.cc:493] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.243580 20205 raft_consensus.cc:3060] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.245697 20227 raft_consensus.cc:2804] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.246327 20227 raft_consensus.cc:493] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.246692 20227 raft_consensus.cc:3060] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.238844 20218 raft_consensus.cc:740] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.251902 20218 consensus_queue.cc:260] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.255023 20218 ts_tablet_manager.cc:1434] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.023s	user 0.010s	sys 0.001s
I20260709 13:06:50.245039 19921 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:50.257689 19921 raft_consensus.cc:2468] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.256485 20218 tablet_bootstrap.cc:492] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.260010 19963 leader_election.cc:304] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.260964 20205 raft_consensus.cc:515] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.261477 20191 raft_consensus.cc:2804] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.262200 20191 raft_consensus.cc:493] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.262645 20191 raft_consensus.cc:3060] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.264854 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:50.265563 19997 raft_consensus.cc:3060] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.258213 20213 leader_election.cc:290] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.273510 19997 raft_consensus.cc:2468] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:50.275540 20036 leader_election.cc:304] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:50.274055 20205 leader_election.cc:290] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:50.276767 20205 raft_consensus.cc:2804] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.277575 20205 raft_consensus.cc:697] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:50.254981 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.278365 20072 raft_consensus.cc:2468] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.281009 20205 consensus_queue.cc:237] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.281988 20227 raft_consensus.cc:515] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.281672 20218 tablet_bootstrap.cc:654] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.277215 20221 ts_tablet_manager.cc:1434] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.081s	user 0.022s	sys 0.016s
I20260709 13:06:50.288404 20221 tablet_bootstrap.cc:492] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.292986 20191 raft_consensus.cc:515] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.295874 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752ecec869a49b79b3c29f06baab9ee" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
W20260709 13:06:50.295984 20037 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.78.65:44645, user_credentials={real_user=slave}} blocked reactor thread for 85281.9us
I20260709 13:06:50.296626 20072 raft_consensus.cc:3060] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.297443 20201 raft_consensus.cc:493] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.297986 20201 raft_consensus.cc:515] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.298961 20221 tablet_bootstrap.cc:654] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.300189 20201 leader_election.cc:290] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.300382 20218 tablet_bootstrap.cc:492] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.300896 20218 ts_tablet_manager.cc:1403] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.045s	user 0.004s	sys 0.008s
I20260709 13:06:50.301059 19921 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:50.301745 19921 raft_consensus.cc:3060] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.294391 20227 leader_election.cc:290] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.294939 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752ecec869a49b79b3c29f06baab9ee" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:50.311906 19997 raft_consensus.cc:3060] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.317992 19997 raft_consensus.cc:2468] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:50.322715 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:50.323414 19923 raft_consensus.cc:3060] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.325723 20227 raft_consensus.cc:493] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.326251 20227 raft_consensus.cc:515] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.327173 19998 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a398a0d465a3459bb48f3564e5a0b87b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:50.328343 19887 leader_election.cc:304] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.328692 20227 leader_election.cc:290] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.324688 20218 raft_consensus.cc:359] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.330300 20071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a398a0d465a3459bb48f3564e5a0b87b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.336202 20071 raft_consensus.cc:2468] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:50.331092 20070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:50.336671 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8415f954b8144792a2ac934b00f876aa" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:50.337153 20070 raft_consensus.cc:3060] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:50.338124 19887 leader_election.cc:343] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:50.331981 20069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8415f954b8144792a2ac934b00f876aa" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.332893 19921 raft_consensus.cc:2468] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 1.
I20260709 13:06:50.319548 20191 leader_election.cc:290] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.327891 19998 raft_consensus.cc:2468] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 0.
I20260709 13:06:50.335502 20201 raft_consensus.cc:2804] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.342571 20201 raft_consensus.cc:697] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:50.343142 19888 leader_election.cc:304] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:50.340718 20072 raft_consensus.cc:2468] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:50.341634 20219 tablet_bootstrap.cc:492] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.352525 20219 ts_tablet_manager.cc:1403] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.143s	user 0.021s	sys 0.023s
I20260709 13:06:50.355861 20219 raft_consensus.cc:359] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.356626 20219 raft_consensus.cc:385] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.343447 20201 consensus_queue.cc:237] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.356927 20219 raft_consensus.cc:740] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.351723 19818 catalog_manager.cc:5697] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 0 to 1, leader changed from <none> to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 1 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:50.349807 20227 raft_consensus.cc:2804] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.339951 20218 raft_consensus.cc:385] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.360354 20227 raft_consensus.cc:493] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.362048 20227 raft_consensus.cc:3060] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.362789 20219 consensus_queue.cc:260] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
W20260709 13:06:50.363641 19888 leader_election.cc:343] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:50.364249 19888 leader_election.cc:304] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:50.360975 20218 raft_consensus.cc:740] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.372941 19820 catalog_manager.cc:5697] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.371001 20201 raft_consensus.cc:2749] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:50.383623 20218 consensus_queue.cc:260] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.387964 20218 ts_tablet_manager.cc:1434] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.087s	user 0.000s	sys 0.011s
I20260709 13:06:50.395233 20218 tablet_bootstrap.cc:492] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.393373 19923 raft_consensus.cc:2468] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.362639 20070 raft_consensus.cc:2468] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.402691 20221 tablet_bootstrap.cc:492] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.403122 19963 leader_election.cc:304] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.403399 20221 ts_tablet_manager.cc:1403] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.115s	user 0.021s	sys 0.030s
I20260709 13:06:50.391494 20227 raft_consensus.cc:515] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.404861 20191 raft_consensus.cc:2804] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.405670 20191 raft_consensus.cc:697] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:50.406576 20191 consensus_queue.cc:237] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.409171 19998 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a398a0d465a3459bb48f3564e5a0b87b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:50.410012 19998 raft_consensus.cc:3060] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.410679 20227 leader_election.cc:290] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.409976 20221 raft_consensus.cc:359] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.411907 20221 raft_consensus.cc:385] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.411836 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a398a0d465a3459bb48f3564e5a0b87b" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:50.412493 20221 raft_consensus.cc:740] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.412706 20072 raft_consensus.cc:3060] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.413482 20221 consensus_queue.cc:260] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.406813 20219 ts_tablet_manager.cc:1434] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.054s	user 0.029s	sys 0.009s
I20260709 13:06:50.417467 20219 tablet_bootstrap.cc:492] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.419322 20218 tablet_bootstrap.cc:654] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.426323 20219 tablet_bootstrap.cc:654] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.429878 20221 ts_tablet_manager.cc:1434] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.003s
I20260709 13:06:50.431078 20221 tablet_bootstrap.cc:492] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.437098 19998 raft_consensus.cc:2468] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:50.438835 19887 leader_election.cc:304] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.438944 20072 raft_consensus.cc:2468] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 1.
I20260709 13:06:50.439985 20227 raft_consensus.cc:2804] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.436237 19819 catalog_manager.cc:5697] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.440562 20227 raft_consensus.cc:697] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:50.441455 20227 consensus_queue.cc:237] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.444555 20221 tablet_bootstrap.cc:654] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.451707 20218 tablet_bootstrap.cc:492] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.453604 20218 ts_tablet_manager.cc:1403] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.059s	user 0.012s	sys 0.005s
I20260709 13:06:50.462916 20191 raft_consensus.cc:493] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.463577 20191 raft_consensus.cc:515] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.464239 20218 raft_consensus.cc:359] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.465526 20218 raft_consensus.cc:385] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.466007 20218 raft_consensus.cc:740] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.467123 20218 consensus_queue.cc:260] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.468143 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de5a97d598a3401da6fdaac95f8c5d0c" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
W20260709 13:06:50.470220 19963 leader_election.cc:343] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:50.470639 20218 ts_tablet_manager.cc:1434] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.000s
I20260709 13:06:50.472364 20218 tablet_bootstrap.cc:492] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.472548 20191 leader_election.cc:290] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.473268 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de5a97d598a3401da6fdaac95f8c5d0c" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.474356 20072 raft_consensus.cc:2468] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.477962 19963 leader_election.cc:304] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:50.479508 20219 tablet_bootstrap.cc:492] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.480331 20219 ts_tablet_manager.cc:1403] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.063s	user 0.018s	sys 0.010s
I20260709 13:06:50.480618 20191 raft_consensus.cc:2804] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.481016 20191 raft_consensus.cc:493] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.481479 20191 raft_consensus.cc:3060] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.484596 20219 raft_consensus.cc:359] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.485719 20219 raft_consensus.cc:385] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.486083 20219 raft_consensus.cc:740] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.487072 20219 consensus_queue.cc:260] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.489730 20219 ts_tablet_manager.cc:1434] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.002s
I20260709 13:06:50.491317 20219 tablet_bootstrap.cc:492] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.493126 20218 tablet_bootstrap.cc:654] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.497779 20221 tablet_bootstrap.cc:492] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.498338 20221 ts_tablet_manager.cc:1403] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.067s	user 0.017s	sys 0.018s
I20260709 13:06:50.498307 19821 catalog_manager.cc:5697] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 1 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.499353 20219 tablet_bootstrap.cc:654] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.501869 20221 raft_consensus.cc:359] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.502729 20221 raft_consensus.cc:385] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.503055 20221 raft_consensus.cc:740] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.503916 20221 consensus_queue.cc:260] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.506347 20221 ts_tablet_manager.cc:1434] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 13:06:50.507512 20221 tablet_bootstrap.cc:492] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.515084 20221 tablet_bootstrap.cc:654] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.518332 20218 tablet_bootstrap.cc:492] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.519137 20218 ts_tablet_manager.cc:1403] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.047s	user 0.017s	sys 0.015s
I20260709 13:06:50.523216 20218 raft_consensus.cc:359] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.524173 20218 raft_consensus.cc:385] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.524514 20218 raft_consensus.cc:740] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.525527 20218 consensus_queue.cc:260] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.535737 20221 tablet_bootstrap.cc:492] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.536442 20221 ts_tablet_manager.cc:1403] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.029s	user 0.008s	sys 0.008s
I20260709 13:06:50.536713 20191 raft_consensus.cc:515] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.539083 20191 leader_election.cc:290] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.540975 20191 raft_consensus.cc:493] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.540858 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de5a97d598a3401da6fdaac95f8c5d0c" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:50.541654 20191 raft_consensus.cc:515] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.542590 20219 tablet_bootstrap.cc:492] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.543255 20219 ts_tablet_manager.cc:1403] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.052s	user 0.024s	sys 0.020s
W20260709 13:06:50.543663 19963 leader_election.cc:343] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:50.543923 20191 leader_election.cc:290] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.542483 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de5a97d598a3401da6fdaac95f8c5d0c" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:50.545912 20070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92383a893f2a4ccfa25834354dd990bf" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.546162 20072 raft_consensus.cc:3060] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.546517 20219 raft_consensus.cc:359] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.547283 20219 raft_consensus.cc:385] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.547703 20219 raft_consensus.cc:740] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
W20260709 13:06:50.548442 19963 leader_election.cc:343] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:50.547595 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92383a893f2a4ccfa25834354dd990bf" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:50.548729 20219 consensus_queue.cc:260] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.550727 20213 raft_consensus.cc:493] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.551608 20219 ts_tablet_manager.cc:1434] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:50.551311 20213 raft_consensus.cc:515] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.552587 20219 tablet_bootstrap.cc:492] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1: Bootstrap starting.
I20260709 13:06:50.553599 20213 leader_election.cc:290] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.549357 19923 raft_consensus.cc:2468] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.555961 20072 raft_consensus.cc:2468] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.558069 20070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.558729 20221 raft_consensus.cc:359] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.559315 20218 ts_tablet_manager.cc:1434] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.040s	user 0.010s	sys 0.000s
I20260709 13:06:50.560405 20219 tablet_bootstrap.cc:654] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.560468 20218 tablet_bootstrap.cc:492] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9: Bootstrap starting.
I20260709 13:06:50.559587 20221 raft_consensus.cc:385] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.562707 20221 raft_consensus.cc:740] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.563560 20221 consensus_queue.cc:260] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.564563 20070 raft_consensus.cc:2468] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.566226 20221 ts_tablet_manager.cc:1434] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.000s
I20260709 13:06:50.567447 20221 tablet_bootstrap.cc:492] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.574687 20221 tablet_bootstrap.cc:654] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.574813 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:50.576071 20218 tablet_bootstrap.cc:654] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.556231 19963 leader_election.cc:304] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:50.576992 19923 raft_consensus.cc:2468] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.579130 20191 raft_consensus.cc:2804] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.579676 20191 raft_consensus.cc:493] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.580148 20191 raft_consensus.cc:3060] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.574819 20219 tablet_bootstrap.cc:492] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1: No bootstrap required, opened a new log
I20260709 13:06:50.582894 20219 ts_tablet_manager.cc:1403] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.003s
I20260709 13:06:50.585847 20219 raft_consensus.cc:359] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.586587 20219 raft_consensus.cc:385] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.586944 20219 raft_consensus.cc:740] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.587801 20219 consensus_queue.cc:260] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.594437 20219 ts_tablet_manager.cc:1434] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:06:50.596998 19963 leader_election.cc:304] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:50.601480 20213 raft_consensus.cc:2804] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.603582 19963 leader_election.cc:304] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:50.604517 20218 tablet_bootstrap.cc:492] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9: No bootstrap required, opened a new log
I20260709 13:06:50.605876 20218 ts_tablet_manager.cc:1403] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent bootstrapping tablet: real 0.046s	user 0.017s	sys 0.005s
I20260709 13:06:50.608825 20218 raft_consensus.cc:359] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.609692 20218 raft_consensus.cc:385] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.609968 20218 raft_consensus.cc:740] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.610776 20218 consensus_queue.cc:260] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.615461 20191 raft_consensus.cc:515] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.617543 20260 raft_consensus.cc:2804] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.623965 20260 raft_consensus.cc:493] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.617076 20213 raft_consensus.cc:697] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:50.625176 20191 leader_election.cc:290] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.624617 20260 raft_consensus.cc:3060] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.626091 20213 consensus_queue.cc:237] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.635041 20213 raft_consensus.cc:493] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.637652 20221 tablet_bootstrap.cc:492] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.638238 20221 ts_tablet_manager.cc:1403] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.071s	user 0.038s	sys 0.020s
I20260709 13:06:50.637972 20213 raft_consensus.cc:515] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.638918 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92383a893f2a4ccfa25834354dd990bf" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:50.641566 20221 raft_consensus.cc:359] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.642320 20221 raft_consensus.cc:385] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.642581 20221 raft_consensus.cc:740] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.643297 20221 consensus_queue.cc:260] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.643388 19821 catalog_manager.cc:5697] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.642724 20213 leader_election.cc:290] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.644835 19921 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8415f954b8144792a2ac934b00f876aa" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:50.645476 20221 ts_tablet_manager.cc:1434] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:06:50.638023 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92383a893f2a4ccfa25834354dd990bf" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:50.649169 19921 raft_consensus.cc:2468] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.649279 19923 raft_consensus.cc:3060] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.649782 20221 tablet_bootstrap.cc:492] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d: Bootstrap starting.
I20260709 13:06:50.650593 19963 leader_election.cc:304] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.647352 20218 ts_tablet_manager.cc:1434] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9: Time spent starting tablet: real 0.041s	user 0.010s	sys 0.026s
I20260709 13:06:50.651702 20263 raft_consensus.cc:2804] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.652125 20263 raft_consensus.cc:493] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.652462 20263 raft_consensus.cc:3060] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.655470 20221 tablet_bootstrap.cc:654] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:50.647075 20260 raft_consensus.cc:515] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.647378 20069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8415f954b8144792a2ac934b00f876aa" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.656311 20069 raft_consensus.cc:2468] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.648381 20072 raft_consensus.cc:3060] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.657783 20260 leader_election.cc:290] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.660377 19921 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:50.661305 19921 raft_consensus.cc:3060] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.663210 19923 raft_consensus.cc:2468] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.665370 20072 raft_consensus.cc:2468] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.667208 19963 leader_election.cc:304] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.683760 20221 tablet_bootstrap.cc:492] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d: No bootstrap required, opened a new log
I20260709 13:06:50.684336 20221 ts_tablet_manager.cc:1403] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d: Time spent bootstrapping tablet: real 0.035s	user 0.018s	sys 0.014s
I20260709 13:06:50.677485 19921 raft_consensus.cc:2468] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.680130 20263 raft_consensus.cc:515] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.666822 20069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:50.687147 20221 raft_consensus.cc:359] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.687898 20069 raft_consensus.cc:3060] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.687922 20221 raft_consensus.cc:385] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:50.688467 20221 raft_consensus.cc:740] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Initialized, Role: FOLLOWER
I20260709 13:06:50.688644 20263 leader_election.cc:290] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.689138 20221 consensus_queue.cc:260] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.690404 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8415f954b8144792a2ac934b00f876aa" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:50.691051 19923 raft_consensus.cc:3060] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.691437 20221 ts_tablet_manager.cc:1434] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:06:50.692735 19963 leader_election.cc:304] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.693240 20070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8415f954b8144792a2ac934b00f876aa" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:50.693974 20070 raft_consensus.cc:3060] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.694538 20263 raft_consensus.cc:2804] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.695065 20263 raft_consensus.cc:697] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:50.696024 20263 consensus_queue.cc:237] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.698827 19923 raft_consensus.cc:2468] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.700188 19963 leader_election.cc:304] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.701128 20213 raft_consensus.cc:2804] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.701771 20213 raft_consensus.cc:697] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:50.702739 20070 raft_consensus.cc:2468] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.702739 20213 consensus_queue.cc:237] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.682968 20260 raft_consensus.cc:2804] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.708345 20260 raft_consensus.cc:697] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:50.709275 20260 consensus_queue.cc:237] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.716167 20069 raft_consensus.cc:2468] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.723125 19820 catalog_manager.cc:5697] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.724231 19820 catalog_manager.cc:5697] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.737862 19820 catalog_manager.cc:5697] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.788923 20205 consensus_queue.cc:1048] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:50.804705 20205 consensus_queue.cc:1048] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:50.831719 20227 consensus_queue.cc:1048] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:50.842476 20245 consensus_queue.cc:1048] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:50.862960 20263 raft_consensus.cc:493] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.863564 20263 raft_consensus.cc:515] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.865237 20263 leader_election.cc:290] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.866125 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d657a45cf2fe4cb592aa1b9c132b519e" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" is_pre_election: true
I20260709 13:06:50.866364 20069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d657a45cf2fe4cb592aa1b9c132b519e" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:50.866923 19923 raft_consensus.cc:2468] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.866963 20069 raft_consensus.cc:2468] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 0.
I20260709 13:06:50.868207 19963 leader_election.cc:304] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:50.868861 20263 raft_consensus.cc:2804] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:50.869212 20263 raft_consensus.cc:493] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:50.869635 20263 raft_consensus.cc:3060] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.874884 20263 raft_consensus.cc:515] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.876783 20263 leader_election.cc:290] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:50.877688 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d657a45cf2fe4cb592aa1b9c132b519e" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:50.877888 20069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d657a45cf2fe4cb592aa1b9c132b519e" candidate_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:50.878417 19923 raft_consensus.cc:3060] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.878484 20069 raft_consensus.cc:3060] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:50.885887 19923 raft_consensus.cc:2468] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.885886 20069 raft_consensus.cc:2468] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in term 1.
I20260709 13:06:50.887320 19963 leader_election.cc:304] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:50.888192 20263 raft_consensus.cc:2804] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:50.888749 20263 raft_consensus.cc:697] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:50.889652 20263 consensus_queue.cc:237] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:50.898231 19821 catalog_manager.cc:5697] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66). New cstate: current_term: 1 leader_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:50.931792 20245 consensus_queue.cc:1048] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:50.946854 20279 consensus_queue.cc:1048] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:50.956203 20263 consensus_queue.cc:1048] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:50.970291 20260 consensus_queue.cc:1048] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:50.998147 20279 raft_consensus.cc:493] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:50.998610 20279 raft_consensus.cc:515] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.000628 20279 leader_election.cc:290] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:51.001396 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de5a97d598a3401da6fdaac95f8c5d0c" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" is_pre_election: true
I20260709 13:06:51.001770 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de5a97d598a3401da6fdaac95f8c5d0c" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74830b4b12e4c6f938dac068fff593d" is_pre_election: true
I20260709 13:06:51.002540 20072 raft_consensus.cc:2393] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in current term 1: Already voted for candidate 2f3794493e4c4d549f357cd6ea8abbf9 in this term.
I20260709 13:06:51.003474 19888 leader_election.cc:304] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:51.004173 20279 raft_consensus.cc:3060] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:51.009653 20279 raft_consensus.cc:2749] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:51.013900 20260 consensus_queue.cc:1048] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.027750 20260 consensus_queue.cc:1048] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.057219 19997 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7"
dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
mode: GRACEFUL
new_leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:49526
I20260709 13:06:51.057914 19997 raft_consensus.cc:606] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Received request to transfer leadership to TS 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:51.063879 20069 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
mode: GRACEFUL
new_leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:40592
I20260709 13:06:51.064476 20069 raft_consensus.cc:606] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Received request to transfer leadership to TS 7c0bca1d38ea415ea1e46ddca89b1fd1
I20260709 13:06:51.088812 19997 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc"
dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
mode: GRACEFUL
new_leader_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:49526
I20260709 13:06:51.089562 19997 raft_consensus.cc:606] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Received request to transfer leadership to TS d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:51.127154 20262 consensus_queue.cc:1048] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.134232 20260 raft_consensus.cc:993] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: : Instructing follower d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:51.134790 20263 raft_consensus.cc:1081] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Signalling peer d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:51.136651 20072 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:40572
I20260709 13:06:51.138681 20072 raft_consensus.cc:493] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:51.139099 20072 raft_consensus.cc:3060] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.140226 20262 consensus_queue.cc:1048] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:51.147035 20072 raft_consensus.cc:515] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.151711 20072 leader_election.cc:290] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 2 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:51.152556 19923 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:51.153308 19923 raft_consensus.cc:3060] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.154877 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f6f7e01e1024f098867a2e2652114bc" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:51.155692 19997 raft_consensus.cc:3055] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:51.156023 19997 raft_consensus.cc:740] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:51.156833 19997 consensus_queue.cc:260] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.157876 19997 raft_consensus.cc:3060] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.159798 19923 raft_consensus.cc:2468] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 2.
I20260709 13:06:51.161038 20037 leader_election.cc:304] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:51.162128 20270 raft_consensus.cc:2804] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:51.162595 20270 raft_consensus.cc:697] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 2 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:51.163501 19997 raft_consensus.cc:2468] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 2.
I20260709 13:06:51.163410 20270 consensus_queue.cc:237] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.170768 19821 catalog_manager.cc:5697] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 1 to 2, leader changed from 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66) to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 2 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:51.191906 20205 raft_consensus.cc:993] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: : Instructing follower 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:51.192451 20205 raft_consensus.cc:1081] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Signalling peer 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:51.193974 19923 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7"
dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:50066
I20260709 13:06:51.194470 19923 raft_consensus.cc:493] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:51.194801 19923 raft_consensus.cc:3060] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.199920 19923 raft_consensus.cc:515] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.201753 19923 leader_election.cc:290] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:51.202558 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:51.203097 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda4eae76b144b33836571c87c8bc0e7" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:51.203130 19997 raft_consensus.cc:3060] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.203652 20072 raft_consensus.cc:3055] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:51.203920 20072 raft_consensus.cc:740] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:51.204593 20072 consensus_queue.cc:260] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.205626 20072 raft_consensus.cc:3060] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.211352 19997 raft_consensus.cc:2468] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:51.211524 20072 raft_consensus.cc:2468] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:51.212533 19887 leader_election.cc:304] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, 7c0bca1d38ea415ea1e46ddca89b1fd1; no voters: 
I20260709 13:06:51.213239 20279 raft_consensus.cc:2804] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:51.213784 20279 raft_consensus.cc:697] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:51.214564 20279 consensus_queue.cc:237] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.218215 20263 consensus_queue.cc:1048] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:51.222929 19821 catalog_manager.cc:5697] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 1 to 2, leader changed from d74830b4b12e4c6f938dac068fff593d (127.7.78.67) to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:51.234629 20263 consensus_queue.cc:1048] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.251312 20263 consensus_queue.cc:1048] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.264400 20260 consensus_queue.cc:1048] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.279565 20260 consensus_queue.cc:1048] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.291446 20213 consensus_queue.cc:1048] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:51.402122 20292 raft_consensus.cc:993] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: : Instructing follower 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:51.402709 20213 raft_consensus.cc:1081] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Signalling peer 7c0bca1d38ea415ea1e46ddca89b1fd1 to start an election
I20260709 13:06:51.404578 19923 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7"
dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
 from {username='slave'} at 127.0.0.1:50076
I20260709 13:06:51.405275 19923 raft_consensus.cc:493] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:51.405706 19923 raft_consensus.cc:3060] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.410996 19923 raft_consensus.cc:515] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.412945 19923 leader_election.cc:290] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Requested vote from peers 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681), d74830b4b12e4c6f938dac068fff593d (127.7.78.67:36893)
I20260709 13:06:51.413767 19997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:51.414124 20072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085ced9b763c4a69a5f0f16de3a724a7" candidate_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
I20260709 13:06:51.414389 19997 raft_consensus.cc:3055] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:51.414733 20072 raft_consensus.cc:3060] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.414769 19997 raft_consensus.cc:740] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:51.415766 19997 consensus_queue.cc:260] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.416658 19997 raft_consensus.cc:3060] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.420398 20072 raft_consensus.cc:2468] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:51.421859 19997 raft_consensus.cc:2468] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0bca1d38ea415ea1e46ddca89b1fd1 in term 2.
I20260709 13:06:51.422053 19888 leader_election.cc:304] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:51.422868 20245 raft_consensus.cc:2804] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:51.423352 20245 raft_consensus.cc:697] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Becoming Leader. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:51.424211 20245 consensus_queue.cc:237] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:51.433238 19821 catalog_manager.cc:5697] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 reported cstate change: term changed from 1 to 2, leader changed from 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66) to 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65). New cstate: current_term: 2 leader_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:51.628705 19998 raft_consensus.cc:1275] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:51.630146 20245 consensus_queue.cc:1048] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:51.640000 20069 raft_consensus.cc:1275] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:51.641860 20201 consensus_queue.cc:1048] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:51.690586 19923 raft_consensus.cc:1275] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:51.691912 20270 consensus_queue.cc:1048] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:51.704869 19998 raft_consensus.cc:1275] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:51.707895 20205 consensus_queue.cc:1048] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:51.933766 19998 raft_consensus.cc:1275] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:51.935102 20279 consensus_queue.cc:1048] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:51.944242 20069 raft_consensus.cc:1275] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Refusing update from remote peer 7c0bca1d38ea415ea1e46ddca89b1fd1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:51.945979 20245 consensus_queue.cc:1048] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:54.093264 19819 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50504:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 13:06:54.119104 19819 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50504:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 13:06:54.178258  7481 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:54.411407 20338 tablet.cc:1724] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.437299 20339 tablet.cc:1724] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.466361 20345 tablet.cc:1724] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.428066 20341 tablet.cc:1724] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.569180 20361 tablet.cc:1724] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:06:54.549067 19887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.66:35681, user_credentials={real_user=slave}} blocked reactor thread for 80400.5us
I20260709 13:06:54.577584 20338 tablet.cc:1724] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.548329 20353 tablet.cc:1724] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.581573 20359 tablet.cc:1724] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.608105 20360 tablet.cc:1724] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.618378 19919 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1"
dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
mode: GRACEFUL
new_leader_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:50088
I20260709 13:06:54.619184 19919 raft_consensus.cc:606] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Received request to transfer leadership to TS d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:54.654407 20339 tablet.cc:1724] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:06:54.638666 19888 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.78.67:36893, user_credentials={real_user=slave}} blocked reactor thread for 99770.7us
I20260709 13:06:54.675302 20345 tablet.cc:1724] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.594879 20362 tablet.cc:1724] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.678505 20365 tablet.cc:1724] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.686829 20366 tablet.cc:1724] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.698586 20364 raft_consensus.cc:993] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1: : Instructing follower d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:54.699270 20279 raft_consensus.cc:1081] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Signalling peer d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:54.698841 20359 tablet.cc:1724] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.700487 20339 tablet.cc:1724] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.713675 20071 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:40586
I20260709 13:06:54.714669 20071 raft_consensus.cc:493] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:54.715183 20071 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.718273 20363 tablet.cc:1724] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.708035 20341 tablet.cc:1724] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.742972 20338 tablet.cc:1724] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.759219 20361 tablet.cc:1724] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.763887 20362 tablet.cc:1724] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.766875 20360 tablet.cc:1724] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.772552 20366 tablet.cc:1724] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.777487 20071 raft_consensus.cc:515] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:54.781278 20071 leader_election.cc:290] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 3 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:54.776574 20353 tablet.cc:1724] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.793099 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:54.793851 19922 raft_consensus.cc:3055] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:54.798400 19922 raft_consensus.cc:740] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7c0bca1d38ea415ea1e46ddca89b1fd1, State: Running, Role: LEADER
I20260709 13:06:54.799243 19922 consensus_queue.cc:260] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:54.800421 19922 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.802310 19997 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336"
dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
mode: GRACEFUL
new_leader_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:51136
I20260709 13:06:54.802973 19997 raft_consensus.cc:606] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 LEADER]: Received request to transfer leadership to TS d74830b4b12e4c6f938dac068fff593d
I20260709 13:06:54.806288  7481 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:54.806945  7481 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:54.807066 20365 tablet.cc:1724] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.809360 19996 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4695d39566674c9a9dadb0c3ef9925f1" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:54.817378 19996 raft_consensus.cc:3060] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.811139 20341 tablet.cc:1724] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.793525 20363 tablet.cc:1724] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:06:54.843490 19996 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 3.
I20260709 13:06:54.843572 19819 catalog_manager.cc:6242] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107: soft_deleted_table [id=252486229c564a4891c8ba398528c33c] alter complete (version 1)
I20260709 13:06:54.853780 19922 raft_consensus.cc:2468] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 3.
I20260709 13:06:54.854161 20036 leader_election.cc:304] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3794493e4c4d549f357cd6ea8abbf9, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:54.855080 20270 raft_consensus.cc:2804] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:54.855513 20270 raft_consensus.cc:697] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 3 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:54.856359 20270 consensus_queue.cc:237] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:54.869946 19820 catalog_manager.cc:5697] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 2 to 3, leader changed from 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65) to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 3 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:55.237443 19998 raft_consensus.cc:1275] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.239027 20270 consensus_queue.cc:1048] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:55.258939 19922 raft_consensus.cc:1275] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.260764 20379 consensus_queue.cc:1048] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:55.298873 20292 raft_consensus.cc:993] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9: : Instructing follower d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:55.299373 20292 raft_consensus.cc:1081] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 LEADER]: Signalling peer d74830b4b12e4c6f938dac068fff593d to start an election
I20260709 13:06:55.301251 20070 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336"
dest_uuid: "d74830b4b12e4c6f938dac068fff593d"
 from {username='slave'} at 127.0.0.1:40572
I20260709 13:06:55.301863 20070 raft_consensus.cc:493] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:55.302237 20070 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:55.308009 20070 raft_consensus.cc:515] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:55.309875 20070 leader_election.cc:290] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 3 election: Requested vote from peers 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645), 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681)
I20260709 13:06:55.310868 19922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1"
I20260709 13:06:55.311152 19998 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c1e2720a76543aaad2d7e42e2c14336" candidate_uuid: "d74830b4b12e4c6f938dac068fff593d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f3794493e4c4d549f357cd6ea8abbf9"
I20260709 13:06:55.311493 19922 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:55.311828 19998 raft_consensus.cc:3055] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:55.312146 19998 raft_consensus.cc:740] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2f3794493e4c4d549f357cd6ea8abbf9, State: Running, Role: LEADER
I20260709 13:06:55.312940 19998 consensus_queue.cc:260] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:55.314036 19998 raft_consensus.cc:3060] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:55.318313 19922 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 3.
I20260709 13:06:55.319681 20037 leader_election.cc:304] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0bca1d38ea415ea1e46ddca89b1fd1, d74830b4b12e4c6f938dac068fff593d; no voters: 
I20260709 13:06:55.320396 20379 raft_consensus.cc:2804] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:55.320751 19998 raft_consensus.cc:2468] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74830b4b12e4c6f938dac068fff593d in term 3.
I20260709 13:06:55.320935 20379 raft_consensus.cc:697] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 3 LEADER]: Becoming Leader. State: Replica: d74830b4b12e4c6f938dac068fff593d, State: Running, Role: LEADER
I20260709 13:06:55.321915 20379 consensus_queue.cc:237] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } }
I20260709 13:06:55.329334 19820 catalog_manager.cc:5697] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d reported cstate change: term changed from 2 to 3, leader changed from 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66) to d74830b4b12e4c6f938dac068fff593d (127.7.78.67). New cstate: current_term: 3 leader_uuid: "d74830b4b12e4c6f938dac068fff593d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74830b4b12e4c6f938dac068fff593d" member_type: VOTER last_known_addr { host: "127.7.78.67" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:55.735249 19922 raft_consensus.cc:1275] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.754024 20270 consensus_queue.cc:1048] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bca1d38ea415ea1e46ddca89b1fd1" member_type: VOTER last_known_addr { host: "127.7.78.65" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:55.778548 19998 raft_consensus.cc:1275] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Refusing update from remote peer d74830b4b12e4c6f938dac068fff593d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.783857 20379 consensus_queue.cc:1048] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3794493e4c4d549f357cd6ea8abbf9" member_type: VOTER last_known_addr { host: "127.7.78.66" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:55.891703  7481 tablet_server.cc:179] TabletServer@127.7.78.65:0 shutting down...
I20260709 13:06:55.921514  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:55.922576  7481 tablet_replica.cc:333] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.923731  7481 raft_consensus.cc:2243] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.924772  7481 raft_consensus.cc:2272] T 0f6f7e01e1024f098867a2e2652114bc P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.929095  7481 tablet_replica.cc:333] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.930437  7481 raft_consensus.cc:2243] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Raft consensus shutting down.
W20260709 13:06:55.930524 20037 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 11 similar messages]
I20260709 13:06:55.932749  7481 raft_consensus.cc:2272] T 6752ecec869a49b79b3c29f06baab9ee P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:55.934922 20037 consensus_peers.cc:597] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d -> Peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Couldn't send request to peer 7c0bca1d38ea415ea1e46ddca89b1fd1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:55.936887  7481 tablet_replica.cc:333] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.937856  7481 raft_consensus.cc:2243] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.938549  7481 raft_consensus.cc:2272] T de5a97d598a3401da6fdaac95f8c5d0c P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.941748  7481 tablet_replica.cc:333] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.942605  7481 raft_consensus.cc:2243] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:55.944836  7481 raft_consensus.cc:2272] T bda4eae76b144b33836571c87c8bc0e7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.948819  7481 tablet_replica.cc:333] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.949612  7481 raft_consensus.cc:2243] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:55.951337  7481 raft_consensus.cc:2272] T b051a1e1b9f04842802d2af08b3bbe9b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.954613  7481 tablet_replica.cc:333] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.955410  7481 raft_consensus.cc:2243] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:55.956503  7481 raft_consensus.cc:2272] T a398a0d465a3459bb48f3564e5a0b87b P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.959584  7481 tablet_replica.cc:333] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.960326  7481 raft_consensus.cc:2243] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.960930  7481 raft_consensus.cc:2272] T 90cbe74d298c4e6c848fab0a23ed0917 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.963912  7481 tablet_replica.cc:333] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.964672  7481 raft_consensus.cc:2243] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.965263  7481 raft_consensus.cc:2272] T 4695d39566674c9a9dadb0c3ef9925f1 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.968948  7481 tablet_replica.cc:333] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.969659  7481 raft_consensus.cc:2243] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.970223  7481 raft_consensus.cc:2272] T 18f26dbbb33b4bc096697663cb163f69 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.972846  7481 tablet_replica.cc:333] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.973515  7481 raft_consensus.cc:2243] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.974013  7481 raft_consensus.cc:2272] T 87c113c7eeba4c09ab636538d2add58a P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.976697  7481 tablet_replica.cc:333] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.977335  7481 raft_consensus.cc:2243] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.977962  7481 raft_consensus.cc:2272] T 32b285df036b4239b948bab122c4b8a5 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.980432  7481 tablet_replica.cc:333] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.981079  7481 raft_consensus.cc:2243] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:55.982460  7481 raft_consensus.cc:2272] T a2c7846e41f249af96a83b97e089f89f P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.985446  7481 tablet_replica.cc:333] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.986092  7481 raft_consensus.cc:2243] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.986711  7481 raft_consensus.cc:2272] T d657a45cf2fe4cb592aa1b9c132b519e P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.989151  7481 tablet_replica.cc:333] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.989754  7481 raft_consensus.cc:2243] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.990403  7481 raft_consensus.cc:2272] T 92383a893f2a4ccfa25834354dd990bf P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.992744  7481 tablet_replica.cc:333] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.993328  7481 raft_consensus.cc:2243] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:55.994465  7481 raft_consensus.cc:2272] T 51d91599e9b540e3a3c7a75ea0d5a338 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:55.997295  7481 tablet_replica.cc:333] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:55.997922  7481 raft_consensus.cc:2243] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:55.998421  7481 raft_consensus.cc:2272] T 5c1e2720a76543aaad2d7e42e2c14336 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.001627  7481 tablet_replica.cc:333] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:56.002271  7481 raft_consensus.cc:2243] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.002945  7481 raft_consensus.cc:2272] T 8415f954b8144792a2ac934b00f876aa P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.006121  7481 tablet_replica.cc:333] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1: stopping tablet replica
I20260709 13:06:56.006747  7481 raft_consensus.cc:2243] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.008427  7481 raft_consensus.cc:2272] T 085ced9b763c4a69a5f0f16de3a724a7 P 7c0bca1d38ea415ea1e46ddca89b1fd1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:56.016285 19963 consensus_peers.cc:597] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 -> Peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Couldn't send request to peer 7c0bca1d38ea415ea1e46ddca89b1fd1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:56.043399 19963 consensus_peers.cc:597] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 -> Peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Couldn't send request to peer 7c0bca1d38ea415ea1e46ddca89b1fd1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.069772  7481 tablet_server.cc:196] TabletServer@127.7.78.65:0 shutdown complete.
I20260709 13:06:56.127199  7481 tablet_server.cc:179] TabletServer@127.7.78.66:0 shutting down...
I20260709 13:06:56.161348  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:56.162359  7481 tablet_replica.cc:333] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.163326  7481 raft_consensus.cc:2243] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.164077  7481 raft_consensus.cc:2272] T 0f6f7e01e1024f098867a2e2652114bc P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.167037  7481 tablet_replica.cc:333] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.167935  7481 raft_consensus.cc:2243] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.168974  7481 raft_consensus.cc:2272] T 6752ecec869a49b79b3c29f06baab9ee P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.173144  7481 tablet_replica.cc:333] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.174206  7481 raft_consensus.cc:2243] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.175181  7481 raft_consensus.cc:2272] T 4695d39566674c9a9dadb0c3ef9925f1 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:56.176048 19963 consensus_peers.cc:597] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 -> Peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Couldn't send request to peer 7c0bca1d38ea415ea1e46ddca89b1fd1. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:44645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.179329  7481 tablet_replica.cc:333] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.180218  7481 raft_consensus.cc:2243] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.180856  7481 raft_consensus.cc:2272] T 18f26dbbb33b4bc096697663cb163f69 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.183715  7481 tablet_replica.cc:333] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.184387  7481 raft_consensus.cc:2243] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.185911  7481 raft_consensus.cc:2272] T 90cbe74d298c4e6c848fab0a23ed0917 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.189302  7481 tablet_replica.cc:333] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.190003  7481 raft_consensus.cc:2243] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.191200  7481 raft_consensus.cc:2272] T 87c113c7eeba4c09ab636538d2add58a P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.194001  7481 tablet_replica.cc:333] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.194722  7481 raft_consensus.cc:2243] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.195858  7481 raft_consensus.cc:2272] T 32b285df036b4239b948bab122c4b8a5 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.199337  7481 tablet_replica.cc:333] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.200171  7481 raft_consensus.cc:2243] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.200847  7481 raft_consensus.cc:2272] T a2c7846e41f249af96a83b97e089f89f P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:56.203547 20037 consensus_peers.cc:597] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d -> Peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Couldn't send request to peer 7c0bca1d38ea415ea1e46ddca89b1fd1. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:44645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.204051  7481 tablet_replica.cc:333] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.204839  7481 raft_consensus.cc:2243] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.205971  7481 raft_consensus.cc:2272] T d657a45cf2fe4cb592aa1b9c132b519e P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.208642  7481 tablet_replica.cc:333] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.209311  7481 raft_consensus.cc:2243] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.210457  7481 raft_consensus.cc:2272] T 92383a893f2a4ccfa25834354dd990bf P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.213068  7481 tablet_replica.cc:333] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.213815  7481 raft_consensus.cc:2243] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.214424  7481 raft_consensus.cc:2272] T 51d91599e9b540e3a3c7a75ea0d5a338 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.217312  7481 tablet_replica.cc:333] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.218056  7481 raft_consensus.cc:2243] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.219084  7481 raft_consensus.cc:2272] T de5a97d598a3401da6fdaac95f8c5d0c P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.221863  7481 tablet_replica.cc:333] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.222499  7481 raft_consensus.cc:2243] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.223250  7481 raft_consensus.cc:2272] T bda4eae76b144b33836571c87c8bc0e7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.226243  7481 tablet_replica.cc:333] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.226857  7481 raft_consensus.cc:2243] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.227348  7481 raft_consensus.cc:2272] T 5c1e2720a76543aaad2d7e42e2c14336 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.231184  7481 tablet_replica.cc:333] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.231900  7481 raft_consensus.cc:2243] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.232539  7481 raft_consensus.cc:2272] T a398a0d465a3459bb48f3564e5a0b87b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.235311  7481 tablet_replica.cc:333] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.235967  7481 raft_consensus.cc:2243] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.236577  7481 raft_consensus.cc:2272] T b051a1e1b9f04842802d2af08b3bbe9b P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.239982  7481 tablet_replica.cc:333] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
I20260709 13:06:56.240751  7481 raft_consensus.cc:2243] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.241986  7481 raft_consensus.cc:2272] T 8415f954b8144792a2ac934b00f876aa P 2f3794493e4c4d549f357cd6ea8abbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.245504  7481 tablet_replica.cc:333] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9: stopping tablet replica
W20260709 13:06:56.245787 20037 consensus_peers.cc:597] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d -> Peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Couldn't send request to peer 7c0bca1d38ea415ea1e46ddca89b1fd1. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:44645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.246284  7481 raft_consensus.cc:2243] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.247408  7481 raft_consensus.cc:2272] T 085ced9b763c4a69a5f0f16de3a724a7 P 2f3794493e4c4d549f357cd6ea8abbf9 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:56.247539 20036 consensus_peers.cc:597] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d -> Peer 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681): Couldn't send request to peer 2f3794493e4c4d549f357cd6ea8abbf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:56.283517 20037 consensus_peers.cc:597] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d -> Peer 7c0bca1d38ea415ea1e46ddca89b1fd1 (127.7.78.65:44645): Couldn't send request to peer 7c0bca1d38ea415ea1e46ddca89b1fd1. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.65:44645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.288125  7481 tablet_server.cc:196] TabletServer@127.7.78.66:0 shutdown complete.
W20260709 13:06:56.316526 20036 consensus_peers.cc:597] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d -> Peer 2f3794493e4c4d549f357cd6ea8abbf9 (127.7.78.66:35681): Couldn't send request to peer 2f3794493e4c4d549f357cd6ea8abbf9. Status: Network error: Client connection negotiation failed: client connection to 127.7.78.66:35681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.333801  7481 tablet_server.cc:179] TabletServer@127.7.78.67:0 shutting down...
I20260709 13:06:56.371613  7481 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:56.372336  7481 tablet_replica.cc:333] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.373139  7481 raft_consensus.cc:2243] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.374181  7481 raft_consensus.cc:2272] T 0f6f7e01e1024f098867a2e2652114bc P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.376798  7481 tablet_replica.cc:333] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.377825  7481 raft_consensus.cc:2243] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.378484  7481 raft_consensus.cc:2272] T 6752ecec869a49b79b3c29f06baab9ee P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.380959  7481 tablet_replica.cc:333] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.381834  7481 raft_consensus.cc:2243] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.382437  7481 raft_consensus.cc:2272] T de5a97d598a3401da6fdaac95f8c5d0c P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.385076  7481 tablet_replica.cc:333] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.385864  7481 raft_consensus.cc:2243] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.386480  7481 raft_consensus.cc:2272] T bda4eae76b144b33836571c87c8bc0e7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.388727  7481 tablet_replica.cc:333] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.389344  7481 raft_consensus.cc:2243] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.389981  7481 raft_consensus.cc:2272] T b051a1e1b9f04842802d2af08b3bbe9b P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.392169  7481 tablet_replica.cc:333] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.392747  7481 raft_consensus.cc:2243] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.393304  7481 raft_consensus.cc:2272] T a398a0d465a3459bb48f3564e5a0b87b P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.395687  7481 tablet_replica.cc:333] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.396273  7481 raft_consensus.cc:2243] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.396765  7481 raft_consensus.cc:2272] T 8415f954b8144792a2ac934b00f876aa P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.398905  7481 tablet_replica.cc:333] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.399457  7481 raft_consensus.cc:2243] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.400020  7481 raft_consensus.cc:2272] T 085ced9b763c4a69a5f0f16de3a724a7 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.402372  7481 tablet_replica.cc:333] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.402973  7481 raft_consensus.cc:2243] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.403506  7481 raft_consensus.cc:2272] T 32b285df036b4239b948bab122c4b8a5 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.405818  7481 tablet_replica.cc:333] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.406383  7481 raft_consensus.cc:2243] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.406848  7481 raft_consensus.cc:2272] T 87c113c7eeba4c09ab636538d2add58a P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.408986  7481 tablet_replica.cc:333] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.409596  7481 raft_consensus.cc:2243] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.410436  7481 raft_consensus.cc:2272] T 18f26dbbb33b4bc096697663cb163f69 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.412479  7481 tablet_replica.cc:333] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.413031  7481 raft_consensus.cc:2243] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.413918  7481 raft_consensus.cc:2272] T 4695d39566674c9a9dadb0c3ef9925f1 P d74830b4b12e4c6f938dac068fff593d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.416195  7481 tablet_replica.cc:333] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.416841  7481 raft_consensus.cc:2243] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.417729  7481 raft_consensus.cc:2272] T 5c1e2720a76543aaad2d7e42e2c14336 P d74830b4b12e4c6f938dac068fff593d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.420506  7481 tablet_replica.cc:333] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.421103  7481 raft_consensus.cc:2243] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.421655  7481 raft_consensus.cc:2272] T d657a45cf2fe4cb592aa1b9c132b519e P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.423872  7481 tablet_replica.cc:333] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.424458  7481 raft_consensus.cc:2243] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.424948  7481 raft_consensus.cc:2272] T 92383a893f2a4ccfa25834354dd990bf P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.427146  7481 tablet_replica.cc:333] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.427652  7481 raft_consensus.cc:2243] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.428134  7481 raft_consensus.cc:2272] T 51d91599e9b540e3a3c7a75ea0d5a338 P d74830b4b12e4c6f938dac068fff593d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.430320  7481 tablet_replica.cc:333] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.430828  7481 raft_consensus.cc:2243] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.431321  7481 raft_consensus.cc:2272] T a2c7846e41f249af96a83b97e089f89f P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.433706  7481 tablet_replica.cc:333] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d: stopping tablet replica
I20260709 13:06:56.434207  7481 raft_consensus.cc:2243] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.434698  7481 raft_consensus.cc:2272] T 90cbe74d298c4e6c848fab0a23ed0917 P d74830b4b12e4c6f938dac068fff593d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.467684  7481 tablet_server.cc:196] TabletServer@127.7.78.67:0 shutdown complete.
I20260709 13:06:56.514168  7481 master.cc:562] Master@127.7.78.126:33377 shutting down...
I20260709 13:06:56.539215  7481 raft_consensus.cc:2243] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.539976  7481 raft_consensus.cc:2272] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.540436  7481 tablet_replica.cc:333] T 00000000000000000000000000000000 P edada50e89b24f0781f15a17e0d28107: stopping tablet replica
I20260709 13:06:56.562667  7481 master.cc:584] Master@127.7.78.126:33377 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (11871 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (24811 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (385044 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 13:06:56.623987  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 411 messages since previous log ~14 seconds ago
I20260709 13:06:56.624248  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 411 messages since previous log ~14 seconds ago
I20260709 13:06:56.624493  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 408 messages since previous log ~14 seconds ago
I20260709 13:06:56.624753  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~33 seconds ago
I20260709 13:06:56.624928  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~33 seconds ago
I20260709 13:06:56.625203  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 3 messages since previous log ~139 seconds ago
I20260709 13:06:56.625483  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 9 messages since previous log ~0 seconds ago
I20260709 13:06:56.625809  7481 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 26 messages since previous log ~6 seconds ago
